Keep asking methods to match wife via the internet? This can be done have got the correct procedure and inspiration. Continue reading for a few important matters you will be doing just before going in online dating services. Primary, make certain you try to get to recognize these kinds of brand new women nicely. Never grow to be also chatty or perhaps also desperate as this can make off of virtually any gal in a very instant. A person must appear assured and comfortable. Second, make use of your own ex-girlfriend latina brides or perhaps ex-wife’s label to start your bill. Thus giving your current account an aura that could be extremely tempting to on the net daters. Additionally, it facilitates as you will find it very difficult to become rejected in case you are not able to contact a friends or family paid members in advance. Eventually, manage several minimal matters although going directly into I have to get long term partner online dating sites. For instance , it is very important you by no means lose your temper or argue using these girls internet. Do not stress and do not speak so that they may must wade through all of the hoke which you set up your profile. No girl loves to be spoken into thus make sure you keep your great while keeping the particular calmness forced to create a very good first impression. Do you wish to know how to connect with better half on-line? Certainly you have to do! So that you can appeal to girls to the dating web site in which you are at present in search of I have to get future spouse internet, you should use the power of your mind. Take the time to loosen up together with meditate from time to time. Hold a positive mindset , nor find carried away from the thrills and even aspiration of actually finding normally the one. Bear in mind the old saying: I have to locate long term spouse internet but it surely is much easier to meet these people online. Make sure that you keep profile neat plus clean without having in to unnecessary talks that just mistake the ladies. This could provide you with the assurance you need and give the possiblity to make an impression these ladies. Be certain that the user profile shouldn’t point out anything phony about you. The account should claim what you would like it to say plus steer clear of offensive feedback. Pursue these basic rules together with soon you should 1000s of flourishing internet dating web-sites through your belt!

Keep asking methods to match wife via the internet? This can be done have got the correct procedure and inspiration. Continue reading for a few important matters you will be doing just before going in online dating services.

Primary, make certain you try to get to recognize these kinds of brand new women nicely. Never grow to be also chatty or perhaps also desperate as this can make off of virtually any gal in a very instant. A person must appear assured and comfortable.

Second, make use of your own ex-girlfriend latina brides or perhaps ex-wife’s label to start your bill. Thus giving your current account an aura that could be extremely tempting to on the net daters. Additionally, it facilitates as you will find it very difficult to become rejected in case you are not able to contact a friends or family paid members in advance.

Eventually, manage several minimal matters although going directly into I have to get long term partner online dating sites. For instance , it is very important you by no means lose your temper or argue using these girls internet. Do not stress and do not speak so that they may must wade through all of the hoke which you set up your profile. No girl loves to be spoken into thus make sure you keep your great while keeping the particular calmness forced to create a very good first impression.

Do you wish to know how to connect with better half on-line? Certainly you have to do!

So that you can appeal to girls to the dating web site in which you are at present in search of I have to get future spouse internet, you should use the power of your mind. Take the time to loosen up together with meditate from time to time. Hold a positive mindset , nor find carried away from the thrills and even aspiration of actually finding normally the one.

Bear in mind the old saying: I have to locate long term spouse internet but it surely is much easier to meet these people online. Make sure that you keep profile neat plus clean without having in to unnecessary talks that just mistake the ladies. This could provide you with the assurance you need and give the possiblity to make an impression these ladies.

Be certain that the user profile shouldn’t point out anything phony about you. The account should claim what you would like it to say plus steer clear of offensive feedback. Pursue these basic rules together with soon you should 1000s of flourishing internet dating web-sites through your belt!

Psychology associated with Gal in Relationships is the next guide I will be looking through and I only cherished it. I was actually capable to examine this mainly because I desired to know exactly how guys can perform with ladies even more efficiently. Within this publication become familiar with how to cope with ladies who wish to consider proper care of the children or even get married. You will additionally learn how you can once the girlfriend fits your needs. Precisely how up to now a girl inside romantic relationships could be the subject matter of this publication. There are several helpful suggestions to follow below. Initially you have to know the type of romantic relationship you desire prior to deciding to possibly think of adult dating somebody. This can be one of the issues with this publication, the in between an informal partnership including a severe romantic relationship. Some other idea I actually appreciated this can be a indisputable fact that you mustn’t go searching to get appreciate within the wrong areas. He claims that if going out there having a young lady that is not really worth their expense it is normally completely wrong of someone to feel that the girl with more than worth it. He likewise https://order-brides.net/ claims you need to get out of typically the rat race and allow your hair down when you are by itself. Despite the fact that he or she in addition mentions that folks have a misconception that will venturing out is usually harmful to your self confidence techniques have a go with going out in addition to carrying out other stuff in addition to working together with spending time with individuals. A final phase belonging to the guide is when the publication talks about the particular psychology associated with lady in romantic relationships. This is a brief phase and share tips approach become a far better lover. Total this guide is very excellent especially if you attempt to learn how to night out a female inside relationships.

Psychology associated with Gal in Relationships is the next guide I will be looking through and I only cherished it. I was actually capable to examine this mainly because I desired to know exactly how guys can perform with ladies even more efficiently. Within this publication become familiar with how to cope with ladies who wish to consider proper care of the children or even get married. You will additionally learn how you can once the girlfriend fits your needs. Precisely how up to now a girl inside romantic relationships could be the subject matter of this publication. There are several helpful suggestions to follow below.

Initially you have to know the type of romantic relationship you desire prior to deciding to possibly think of adult dating somebody. This can be one of the issues with this publication, the in between an informal partnership including a severe romantic relationship. Some other idea I actually appreciated this can be a indisputable fact that you mustn’t go searching to get appreciate within the wrong areas. He claims that if going out there having a young lady that is not really worth their expense it is normally completely wrong of someone to feel that the girl with more than worth it. He likewise https://order-brides.net/ claims you need to get out of typically the rat race and allow your hair down when you are by itself. Despite the fact that he or she in addition mentions that folks have a misconception that will venturing out is usually harmful to your self confidence techniques have a go with going out in addition to carrying out other stuff in addition to working together with spending time with individuals.

A final phase belonging to the guide is when the publication talks about the particular psychology associated with lady in romantic relationships. This is a brief phase and share tips approach become a far better lover. Total this guide is very excellent especially if you attempt to learn how to night out a female inside relationships.

Mindset involving Lady within Relationships certainly is the following guide We are looking at and I simply just adored that. I used to be actually capable to go through it since I wanted to know precisely how adult men can work along with ladies even more successfully. From this publication become familiar with how to deal with girls that wish to consider care of your children or even marry. Additionally, you will study what you’ll do once the woman fits your needs. Exactly how https://bridesclub.org/latin/cuban-brides/ as of yet a girl in human relationships would be the area of interest of this publication. There are several beneficial guidelines to go along with below. Very first you have to know the sort of relationship you would like prior to deciding to even think about internet dating an individual. This can be among the issues during this publication, the difference between a casual relationship and a really serious romantic relationship. A further idea I just loved the idea that you must not search just for really enjoy within the incorrect spots. He claims when you decide to go out there using a female who will be certainly not worth the cost it might be wrong regarding you to definitely think that completely of great benefit. They in addition states that you should step out of typically the rat race and let your hair down while you are alone. Despite the fact that he or she as well describes that people have a misunderstanding that will heading out is normally detrimental to your own confidence so do try out going out plus undertaking other things rather than working plus spending some time with people. One more phase on the book is where typically the publication talks about typically the psychology involving gal within romantic relationships. It is just a short part and gives some guidelines for you to be a much better fan. Total this guide is actually great especially if you are trying to figure out how to time a woman inside romances.

Mindset involving Lady within Relationships certainly is the following guide We are looking at and I simply just adored that. I used to be actually capable to go through it since I wanted to know precisely how adult men can work along with ladies even more successfully. From this publication become familiar with how to deal with girls that wish to consider care of your children or even marry. Additionally, you will study what you’ll do once the woman fits your needs. Exactly how https://bridesclub.org/latin/cuban-brides/ as of yet a girl in human relationships would be the area of interest of this publication. There are several beneficial guidelines to go along with below.

Very first you have to know the sort of relationship you would like prior to deciding to even think about internet dating an individual. This can be among the issues during this publication, the difference between a casual relationship and a really serious romantic relationship. A further idea I just loved the idea that you must not search just for really enjoy within the incorrect spots. He claims when you decide to go out there using a female who will be certainly not worth the cost it might be wrong regarding you to definitely think that completely of great benefit. They in addition states that you should step out of typically the rat race and let your hair down while you are alone. Despite the fact that he or she as well describes that people have a misunderstanding that will heading out is normally detrimental to your own confidence so do try out going out plus undertaking other things rather than working plus spending some time with people.

One more phase on the book is where typically the publication talks about typically the psychology involving gal within romantic relationships. It is just a short part and gives some guidelines for you to be a much better fan. Total this guide is actually great especially if you are trying to figure out how to time a woman inside romances.

如何利用软件定义简化网络运营维护工作

一、为什么是软件定义?

在这个“软件定义一切”的时代(有人戏称为“SDx”),网络自然也是其涵盖的范畴之一,应运而生的SDN便是一个方向,至于如何理解SDN,笔者更愿从广义的角度来看,而不单单是其狭义定义中的三个特点,即:控转分离、集中控制、开放可编程。或者说狭义的SDN是为广义的SDN来服务的,对底层网络的变革,带来的更大的好处是在后期网络运营时,对流量的可视化、流量调度、监控报警等方面带来更多的可操作性、可实施性及便捷性。

二、网络运营会面临哪些问题?

当一张网络在线运行之后,技术人员更多的时间和精力会投入到网络的优化、变更、监控、报警等,总结起来几个方面:
– 可视化
– 操作简化
– 实时监控
– 快速定位
– 可感知

面临大量维护工作的同时,还要总结网络是否满足日益繁杂的业务要求,是否需要调整,是否需要优化,如何优化,如何规避等问题,这是迭代过程中的必然过程,正所谓“打江山容易,守江山难”。

三、传统网络技术环境有哪些弊端?

面临网络运营问题,传统网络技术环境多为网络设备“堆砌”而成,“盒子”由设备厂商所有,很难看到一个全量的、透明的网络世界,因此在实现这些网络运营需求的过程中,技术人员只能通过标准协议将有限的MIB信息读取至各种或开源或商业化的网络信息平台,从而实现有限程度的运营需求,究其原因,笔者认为主要集中在以下几点:
– 厂商私有化严重,相对闭源。
– 诸如snmp、netconf等标准化协议的限制,无法更好的满足技术需求。
– 各种网络工具软件灵活性不够、功能性不足、适配性欠缺,需要定制或二次开发,部署难度大。

网络从业人员在面临上述问题时,往往为了网络运营这一刚需,会搭建大量相对无关联的网络信息平台、自定义监控脚本,费时费力的同时也只能满足部分监控指标,且这些平台本身也需要技术人员不定期维护。

四、如何给网络“松绑”?

当前网络技术发展处于开源与闭源的过渡期,之所以这么说,是因为以SDN为代表新型网络技术还没有完全落地,与之对应的硬件产品也在市场化过程中,且需要市场的考虑及技术的沉淀,而以传统网络厂商为代表的闭源产品,也正在融入开放、开源的理念在其产品中,很多网络设备已支持OpenFlow等南向协议、开放gRPC接口,甚至你可以把一台交换机视作一台服务器,可在其本地编写/运行脚本,这些改善有助于在网络技术迭代的过程中,完善、加强网络运营指标,更好服务于业务需求。笔者这里提供几点部署建议,供参考:
– 在保留既有网络投入及网络架构的前提下,引入网络开源协议或技术。
– 将新老技术混合应用在网络环境中,这并不冲突。
– 尝试“过渡式”的技术实现方案,新老技术并行,如:传统网络技术承载业务,开源网络方案介入网络管理。

五、网络平稳过渡很重要

网络平稳的重要性不言而喻,网络作为所有业务生产者/消费者的“传输媒介”,其稳定性将直接影响产品体验,新技术的引入、网络的迭代一定要与旧技术“并行”,在不影响当前业务的前提下,改善网络运营环境。示例如下图:

  1. 网络设备通过OpenFlow、gRPC等将管理流量引入SDN控制端,通过面向北向接口的各种平台多维度分析(所见即所得)。
  2. 对具备研发能力的网络人员可基于开源的SDN CONTROLLER(ODL/ONOS)定制网络模型、流量分类、数据标记等。
  3. 与三方的开源网络平台对接,如API等方式,摆脱传统的只能SNMP上报流量的方式。

六、小结

网络新型技术的出现必将改变传统的网络实现方式,并对“网络世界”的视角变得更加全面,流量见得大而全,数据才能分析的透,这对网络更全面的优化、了解流量更全面的特点、深耕业务的发展方向提供更有意义的指导和建议。

TIDB在伴鱼的实践

一、背景介绍

伴鱼少儿英语是目前飞速成长的互联网在线英语教育品牌之一,旗下包括伴鱼绘本、伴鱼少儿英语、伴鱼自然拼读和伴鱼精读课等系列产品。伴鱼使用的数据库种类有TiDB,MongoDB,MySQL,Codis。在伴鱼,目前生产环境共有10套TiDB集群,服务于在线教学、绘本、消息、交易和少儿等众多核心业务。

目前线上Tidb部署版本统一为2.1.15。TiDB server和PD混部,机器配置:64C64G500G SAS;Tikv单独部署,机器配置:64C64G1.5T NVME

下面从伴鱼为何选择TiDB,伴鱼TiDB架构、伴鱼在使用TiDB过程中遇到的问题、我们是如何解决这些问题的以及TiDB在伴鱼的后续使用计划等几个方面进行阐述。

二、伴鱼为何选择TiDB

伴鱼发展之初,选择了MongoDB作为数据存储,但随着业务的快速发展,逐渐暴露出一些问题,比如业务对事务的需求、单表数据极速增长导致的性能问题以及集群实例容量问题等。因此,我们对新存储的诉求有以下几点:

集群高可用
MongoDB副本集架构支持高可用,不需要通过第三方程序来支持高可用,可以减少依赖和运维成本。因此对于新存储,我们也需要它天然支持高可用。

支持事务
目前我们生产使用的MongoDB 3.2版本对事务的支持较弱,而业务的很多场景需要事务的支持。

大容量、高吞吐
MongoDB副本集主节点永远只有一个,当业务快速发展时,可能出现写瓶颈,如果这时想扩展写就非常痛苦了。同时,机器的磁盘空间有限,在数据增长到一定量级时,也会出现集群容量瓶颈的问题。

不分表
不管MySQL还是MongoDB数据库,表数据达到一定的量级,读写性能都会下降。通过分表可以提高读写性能,但是业务代码需要维护比较复杂的分表逻辑和路由。

基于以上这些需求,我们通过调研,选择了当前流行的TiDB分布式数据库,主要原因包括:TiDB天然支持以上几点需求;同时TiDB已经在很多公司使用起来,经历了众多业务场景的历练,而且我们的业务使用场景也比较类似。

三、伴鱼TiDB架构

TiDB 集群主要包括三个核心组件:TiDB Server,PD Server 和 TiKV Server。

TiDB Server
TiDB Server负责接收SQL请求,处理SQL相关的逻辑,并通过PD找到存储计算所需数据的TiKV地址,与TiKV交互获取数据,最终返回结果。TiDB Server是无状态的,其本身并不存储数据,只负责计算,可以无限水平扩展。

PD Server
Placement Driver是整个集群的管理模块,其主要工作有三个:一是存储集群的元信息(某个Key存储在哪个TiKV节点);二是对TiKV集群进行调度和负载均衡(如数据的迁移、Raft group leader的迁移等);三是分配全局唯一且递增的事务ID。
PD通过Raft协议保证数据的安全性。Raft的leader server负责处理所有操作,其余的PD server仅用于保证高可用。

TiKV Server
TiKV Server负责存储数据,从外部看TiKV是一个分布式的提供事务的Key-Value存储引擎。存储数据的基本单位是Region,每个Region负责存储一个Key Range(从 StartKey到EndKey的左闭右开区间)的数据,每个TiKV节点会负责多个Region。TiKV使用Raft协议做复制,保持数据的一致性和容灾。副本以Region为单位进行管理,不同节点上的多个Region构成一个Raft Group,互为副本。数据在多个TiKV之间的负载均衡由PD调度,这里也是以Region为单位进行调度。

伴鱼TiDB架构
目前伴鱼Tidb的业务接入方式主要通过SLB接入,当TiDB-Server节点宕机,10s可以被负载均衡识别,自动剔除故障节点。PD和Tikv数据默认三副本,组件自身的高可用由raft算法协议保证。

四、伴鱼在使用TiDB过程中遇到的问题
伴鱼在使用TiDB之初,我们统一了集群使用的TiDB版本,规范了版本的配置,这样可以避免维护多个版本带来的潜在问题,同时有利于增强我们对特定版本的把控程度。我们生产环境的TiDB版本为2.1.15,随着业务的快速发展,主要遇到以下几个问题:

优化器选择索引不准确
在生产环境中,我们碰到过以下几种现象
1)单表数据30W+,查询请求并发约10+,某次业务上线,新增一个索引后,导致原有的查询索引选择错误,tikv实例所在机器cpu迅速被打满,引发故障。
2)线上某张大表,请求量比较大,偶尔出现个别条件走不到索引,导致全表扫描,从而引发接口响应时间的抖动,影响业务。
3)线上某张14亿的大表,查询条件区分度很高,某天出现特定条件突然走不到索引,导致全表扫描,引发故障。后面经过pingcap同学排查,系bug导致。

数据库性能问题定位慢
生产环境发生过一次今人印象深刻的故障,持续时间约一天。其实现在回过头来看,问题可以简单描述为:一个sql由于走错索引,数据扫描比较大,同时加上高峰期并发比较大,导致tikv cpu被打满,导致整个集群响应时间变高,引发故障。但是这次问题的定位,我们付出了很大的代价。

大数据同步问题
许多公司都有数据分析的需求,我们把上游各Tidb集群的数据通过Pump/Drainer汇聚到一个TiDB集群供大数据分析使用。在使用过程中,遇到数据不一致以及数据同步慢等问题。

五、我们如何解决在使用TiDB过程中遇到的问题

对于优化器选择索引不准确问题
在开发使用TiDB的过程中,对于数据量大的表和请求量比较大的表,我们强烈要求开发使用强制索引,防止在特殊情况下,请求走不到索引。同时,我们SQL审核平台,对新建表的索引数量都有严格的限制,防止研发同学建过多的索引。索引越少,理论上可以降低优化器选错索引的概率。

对于数据库性能问题的快速定位
TiDB兼容了MySQL协议,运维方式上跟MySQL有很多相似之处。我们认为,数据库的性能问题,绝大部分原因都是由慢SQL导致的,当然像数据库bug、业务异常流量等情况,在伴鱼还是比较少见的。所以,我们如何准实时收集分布式TiDB的慢日志、如何快速的做分析统计以及如何及时的告警,对于快速解决线上问题,甚至将性能风险扼杀在摇篮里至关重要。

1)如何快速收集TiDB慢日志
我们采用了业界比较成熟的开源日志采集/分析/存储架构,很好的解决了我们对慢日志的分析统计需求。

日志采集架构中,其中filebeat负责增量收集TiDB产生的慢日志,由于filebeat比较轻量,对线上性能基本无影响;kafka负责接收从filebeat采集过来的慢日志;logstash负责读取kafka中的慢日志并进行解析,转换成我们想要的kv键值对;最后解析后的数据入到es,供kibana查询分析统计。

对于Tidb的慢日志,我们重点关注慢日志中的某些特定字段,比如:
Time:表示日志打印时间。
DB:表示执行语句时使用的database。
Query_time:表示执行这个语句花费的时间。
Total_keys:表示Coprocessor扫过的key的数量。
Process_keys:表示Coprocessor处理的key的数量。相比total_keys,processed_keys不包含 MVCC的旧版本。
SQL:执行的sql语句。

2)如何快速的分析统计
logstash解析的数据入到es,通过kibana,我们可以很方便的对数据进行分析统计,比如:
1)近5分钟内,查询时间超过1s的请求/倒序排列/按表统计
2)近5分钟内,Process_keys大于5000的请求/倒序排列/按表统计
3)按照业务db统计慢日志数量
等等,同时我们还开发了慢日志分析报表平台,从各个维度(库/表/操作类型/慢日志数量/总时间/平均响应时间等)对慢日志进行统计分析,及时发现性能风险。

3)如何及时的告警
在伴鱼,一个db对应一个服务,所以告警都是在特定db下设置规则。目前,我们告警粒度是一分钟,主要基于以下三类规则告警:某个db的慢日志达到一定数量则告警;某个db下的请求时间超过500ms且达到一定数量则告警;某个db下的查询Process_keys大于1000且达到一定数量则告警。当然告警规则的设置不是一蹴而就的,需要根据不同的业务场景,不断的调整,最终达到一个比较合理的阀值。

对于大数据同步问题
我们通过pump/drainer将上游Tidb的数据同步到大数据Tidb集群,供大数据分析统计。在使用过程中,我们遇到以下几个问题,并逐一解决。
1)某天业务反馈,大数据集群某些表的数据少于线上集群。通过排查发现,上游某张表数据乱码,drainer同步下游报错,导致服务重启,重启时同步的位点不正确,导致上下游数据不正确。最后通过修复上游数据,重新同步解决。
2)drainer消费数据很慢,通过加大txn-batch/worker-count数量解决。
3)TiDB大数据集群raft store cpu长期居高临下,我们通过开启开启Region Merge和调整raft-base-tick-interval解决,最终raft store cpu利用率从90%下降到50%。

六、TiDB在伴鱼的后续展望
目前,我们在保证生产环境TiDB-2.1.15现有版本的稳定的同时,也在调研Tidb-3.0.x新版本的一些特性。从官方介绍,我们重点关注以下特性:
1)3.0与2.1版本相比,提升了大规模集群的稳定性。优化Raft副本之间的心跳机制,按照Region的活跃程度调整心跳频率,减小冷数据对集群的负担;热点调度策略支持更多参数配置,采用更高优先级,并提升热点调度的准确性;优化PD调度流程,提供调度限流机制;新增分布式GC 功能,提升GC的性能,降低大集群GC时间。
2)TiDB 3.0版本采用多种优化手段提升查询计划的稳定性。如新增Fast Analyze功能,提升收集统计信息的速度,降低集群资源的消耗及对业务的影响;新增SQL Plan Management功能,支持在查询计划不准确时手动绑定查询计划。
3)3.0与2.1版本相比,在查询性能上有很大的提升。如TiDB持续优化SQL执行器;TiKV新增多线程Raftstore和Apply功能,提升单节点内并发处理能力和资源利用率,降低延时,大幅提升集群写入能力。
因此,我们计划在2020年上半年,陆续将生产Tidb版本从2.1升级到3.0版本。

Kubernetes容器网络

Kubernetes容器网络

在kubernetes中要保证容器之间网络互通,网络至关重要.而kubernetes本身并没有具体实现网络,而是通过插件化的方式自由接入。在容器网络接入进来需要满足基本原则:

  • pod无论运行在任何节点都可以互相直接通信,而不需要借助NAT地址转换实现。
  • node与pod可以互相通信,在不限制的前提下,pod可以访问任意网络。
  • pod拥有独立的网络栈,pod看到自己的地址和外部看见的地址应该是一样的,并且同个pod内所有的容器共享同个网络栈。

容器网络基础

一个Linux容器的网络栈是被隔离在它自己的Network Namespace中,Network Namespace包括了:网卡(Network Interface),回环设备(Lookback Device),路由表(Routing Table)和iptables规则,对于服务进程来讲这些就构建了它发起请求和相应的基本环境。而要实现一个容器网络,离不开以下linux网络功能:

  • 网络命名空间: 将独立的网络协议栈隔离到不同的命令空间中,彼此间无法通信。
  • Veth Pair: Veth设备对的引入是为了实现在不同网络命名空间的通信,总是以两张虚拟网卡(veth peer) 的形式成对出现的。并且,从其中一端发出的数据,总是能在另外一端收到。
  • Iptables/Netfilter: Netfilter负责在内核中执行各种定义的规则(过滤、修改、丢弃等),运行在内核中;Iptables模式是在用户模式下运行的进程,负责协助维护内核中Netfilter的各种规则表;通过二者的配合来实现整个Linux网络协议栈中灵活的数据包处理机制
  • 网桥: 网桥是一个二层网络虚拟设备,类似交换机,主要功能是通过学习而来的Mac地址将数据帧转发到网桥的不同端口上.

  • 路由: Linux系统包含一个完整的路由功能,当IP层在处理数据发送或转发的时候,会使用路由表来决定发往哪里。

基于以上的基础,同宿主机的容器时间如何通信呢?
我们可以简单把他们理解成两台主机,主机之间通过网线连接起来,如果要多台主机通信,我们通过交换机就可以实现彼此互通。在容器中,以上的实现是通过docker0网桥,凡是连接到docker0网桥的容器,就可以通过它来进行通信。要想容器能够连接到docker0网桥,我们只需要类似网线的虚拟设备Veth Pair来把容器连接到网桥上即可。

我们启动一个容器:

$ docker run -d --name c1  hub.pri.ibanyu.com/devops/alpine:v3.8 /bin/sh

然后查看网卡设备:

$ docker exec -it c1  /bin/sh
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1172 (1.1 KiB)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.17.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0

可以看到其中有一张eth0的网卡,它就是veth peer其中的一端的虚拟网卡。然后通过route -n 查看容器中的路由表,eth0也正是默认路由出口。所有对172.17.0.0/16网段的请求都会从eth0出去。
我们再来看Veth peer的另一端,我们查看宿主机的网络设备:

# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:6aff:fe46:93d2  prefixlen 64  scopeid 0x20<link>
        ether 02:42:6a:46:93:d2  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 656 (656.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.100.0.2  netmask 255.255.255.0  broadcast 10.100.0.255
        inet6 fe80::5400:2ff:fea3:4b44  prefixlen 64  scopeid 0x20<link>
        ether 56:00:02:a3:4b:44  txqueuelen 1000  (Ethernet)
        RX packets 7788093  bytes 9899954680 (9.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5512037  bytes 9512685850 (8.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 32  bytes 2592 (2.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 32  bytes 2592 (2.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth20b3dac: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::30e2:9cff:fe45:329  prefixlen 64  scopeid 0x20<link>
        ether 32:e2:9c:45:03:29  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 656 (656.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

我们可以看到,容器对应的Veth peer 另一端是宿主机上的一块虚拟网卡叫veth20b3dac,并且可以通过brctl 查看网桥信息看到这张网卡是在docker0上。

# brctl show
docker0     8000.02426a4693d2   no      veth20b3dac

然后我们再启动一个容器,从第一个容器是否能ping通第二个容器.

$ docker run -d --name c2 hub.pri.ibanyu.com/devops/alpine:v3.8 /bin/sh
$ docker exec -it c1 /bin/sh
/ # ping 172.17.0.3
PING 172.17.0.3 (172.17.0.3): 56 data bytes
64 bytes from 172.17.0.3: seq=0 ttl=64 time=0.291 ms
64 bytes from 172.17.0.3: seq=1 ttl=64 time=0.129 ms
64 bytes from 172.17.0.3: seq=2 ttl=64 time=0.142 ms
64 bytes from 172.17.0.3: seq=3 ttl=64 time=0.169 ms
64 bytes from 172.17.0.3: seq=4 ttl=64 time=0.194 ms
^C
--- 172.17.0.3 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.129/0.185/0.291 ms

可以看到,能够ping通,其原理就是我们ping 目标IP172.17.0.3时,会匹配到我们的路由表第二条规则,网关为0.0.0.0,这就意味着是一条直连路由,通过二层转发到目的地。要通过二层网络到达172.17.0.3,我们需要知道它的Mac地址,此时就需要第一个容器发送一个ARP广播,来通过IP地址查找它的mac地址。此时Veth peer另外一段是docker0网桥,它会广播到所有连接它的veth peer 虚拟网卡去,然后正确的虚拟网卡收到后会响应这个ARP报文,然后网桥再回给第一个容器.
以上就是同宿主机不同容器通过docker0通信,如下图所示:

默认情况下,通过network namespace限制的容器进程,本质上是通过Veth peer设备和宿主机网桥的方式,实现了不同network namespace 的数据交换。

与之类似地,当你在一台宿主机上,访问该宿主机上的容器的 IP 地址时,这个请求的数据包,也是先根据路由规则到达 docker0 网桥,然后被转发到对应的 Veth Pair 设备,最后出现在容器里。

跨主机网络通信

在 Docker 的默认配置下,不同宿主机上的容器通过 IP 地址进行互相访问是根本做不到的。为了解决这个问题,社区中出现了很多网络方案。同时k8s为了更好的控制网络的接入,推出了CNI即容器网络的API接口。它是k8s中标准的一个调用网络实现的接口,kubelet通过这个API来调用不同的网络插件以实现不同的网络配置,实现了这个接口的就是CNI插件,它实现了一系列的CNI API接口。目前已经有的包括flannel、calico、weave、contiv等等。

实际上CNI的容器网络通信流程跟前面的基础网络一样,只是CNI维护了一个单独的网桥来代替 docker0。这个网桥的名字就叫作:CNI 网桥,它在宿主机上的设备名称默认是:cni0。cni的设计思想就是:Kubernetes 在启动 Infra 容器之后,就可以直接调用 CNI 网络插件,为这个 Infra 容器的 Network Namespace配置符合预期的网络栈。

CNI插件三种网络实现模式:

  • overlay 模式是基于隧道技术实现的,整个容器网络和主机网络独立,容器之间跨主机通信时将整个容器网络封装到底层网络中,然后到达目标机器后再解封装传递到目标容器。不依赖与底层网络的实现。实现的插件有flannel(UDP、vxlan)、calico(IPIP)等等

  • 三层路由模式中容器和主机也属于不通的网段,他们容器互通主要是基于路由表打通,无需在主机之间建立隧道封包。但是限制条件必须依赖大二层同个局域网内。实现的插件有flannel(host-gw)、calico(BGP)等等

  • underlay网络是底层网络,负责互联互通。 容器网络和主机网络依然分属不同的网段,但是彼此处于同一层网络,处于相同的地位。整个网络三层互通,没有大二层的限制,但是需要强依赖底层网络的实现支持.实现的插件有calico(BGP)等等

我们看下路由模式的一种实现flannel Host-gw:

如图可以看到当node1上container-1要发数据给node2上的container2时,会匹配到如下的路由表规则:

10.244.1.0/24 via 10.168.0.3 dev eth0

表示前往目标网段10.244.1.0/24的IP包,需要经过本机eth0出去发往的下一跳ip地址为10.168.0.3(node2).然后到达10.168.0.3以后再通过路由表转发到cni网桥,进而进入到container2。

以上可以看到host-gw工作原理,其实就是在每个node节点配置到每个pod网段的下一跳为pod网段所在的node节点IP,pod网段和node节点ip的映射关系,flannel保存在etcd或者k8s中。flannel只需要watch 这些数据的变化来动态更新路由表即可.

这种网络模式最大的好处就是避免了额外的封包和解包带来的网络性能损耗。缺点我们也能看见主要就是容器ip包通过下一跳出去时,必须要二层通信封装成数据帧发送到下一跳。如果不在同个二层局域网,那么就要交给三层网关,而此时网关是不知道目标容器网络的(也可以静态在每个网关配置pod网段路由)。所以flannel host-gw必须要求集群宿主机是二层互通的。

而为了解决二层互通的限制性,calico提供的网络方案就可以更好的实现,calico 大三层网络模式与flannel 提供的类似,也会在每台宿主机添加如下格式的路由规则:

<目标容器IP网段> via <网关的IP地址> dev eth0

其中网关的IP地址在不同场景有不同的意思,如果宿主机是二层可达那么就是目的容器所在的宿主机的IP地址,如果是三层不同局域网那么就是本机宿主机的网关IP(交换机或者路由器地址).

不同于flannel通过k8s或者etcd存储的数据来维护本机路由信息的做法,calico是通过BGP动态路由协议来分发整个集群路由信息。

BGP全称是 Border Gateway Protocol边界网关协议,linxu原生支持的、专门用于在大规模数据中心为不同的自治系统之间传递路由信息。只要记住BGP简单理解其实就是实现大规模网络中节点路由信息同步共享的一种协议.而BGP这种协议就能代替flannel 维护主机路由表功能。

calico 主要由以下几个部分组成:

  • calico cni插件: 主要负责与kubernetes对接,供kubelet调用使用。
  • felix: 负责维护宿主机上的路由规则、FIB转发信息库等.
  • BIRD: 负责分发路由规则,类似路由器.
  • confd: 配置管理组件。

除此之外,calico还和flannel host-gw不同之处在于,它不会创建网桥设备,而是通过路由表来维护每个pod的通信,如下图所示:

可以看到calico 的cni插件会为每个容器设置一个veth pair设备,然后把另一端接入到宿主机网络空间,由于没有网桥,cni插件还需要在宿主机上为每个容器的veth pair设备配置一条路由规则,用于接收传入的IP包,路由规则如下:

10.92.77.163 dev cali93a8a799fe1 scope link

以上表示发送10.92.77.163的IP包应该发给cali93a8a799fe1设备,然后到达另外一段容器中。

有了这样的veth pair设备以后,容器发出的IP包就会通过veth pair设备到达宿主机,然后宿主机根据路由规则的下一条地址,发送给正确的网关(10.100.1.3),然后到达目标宿主机,在到达目标容器.

10.92.77.0/24 via 10.100.1.3 dev bond0 proto bird

这些路由规则都是felix维护配置的,而路由信息则是calico bird组件基于BGP分发而来。calico实际上是将集群里所有的节点都当做边界路由器来处理,他们一起组成了一个全互联的网络,彼此之间通过BGP交换路由,这些节点我们叫做BGP Peer。

需要注意的是calico 维护网络的默认模式是 node-to-node mesh ,这种模式下,每台宿主机的BGP client都会跟集群所有的节点BGP client进行通信交换路由。这样一来,随着节点规模数量N的增加,连接会以N的2次方增长,会集群网络本身带来巨大压力。
所以一般这种模式推荐的集群规模在50节点左右,超过50节点推荐使用另外一种RR(Router Reflector)模式,这种模式下,calico 可以指定几个节点作为RR,他们负责跟所有节点BGP client建立通信来学习集群所有的路由,其他节点只需要跟RR节点交换路由即可。这样大大降低了连接数量,同时为了集群网络稳定性,建议RR>=2.

以上的工作原理依然是在二层通信,当我们有两台宿主机,一台是10.100.0.2/24,节点上容器网络是10.92.204.0/24;另外一台是10.100.1.2/24,节点上容器网络是10.92.203.0/24,此时两台机器因为不在同个二层所以需要三层路由通信,这时calico就会在节点上生成如下路由表:

10.92.203.0/24 via 10.100.1.2 dev eth0 proto bird

这时候问题就来,因为10.100.1.2跟我们10.100.0.2不在同个子网,是不能二层通信的。这之后就需要使用Calico IPIP模式,当宿主机不在同个二层网络时就是用overlay网络封装以后再发出去。如下图所示:

IPIP模式下在非二层通信时,calico 会在node节点添加如下路由规则:

10.92.203.0/24 via 10.100.1.2 dev tunnel0

可以看到尽管下一条任然是node的IP地址,但是出口设备却是tunnel0,其是一个IP隧道设备,主要有Linux内核的IPIP驱动实现。会将容器的ip包直接封装宿主机网络的IP包中,这样到达node2以后再经过IPIP驱动拆包拿到原始容器IP包,然后通过路由规则发送给veth pair设备到达目标容器。

以上尽管可以解决非二层网络通信,但是仍然会因为封包和解包导致性能下降。如果calico 能够让宿主机之间的router设备也学习到容器路由规则,这样就可以直接三层通信了。比如在路由器添加如下的路由表:

10.92.203.0/24 via 10.100.1.2 dev interface1

而node1添加如下的路由表:

10.92.203.0/24 via 10.100.1.1 dev tunnel0

那么node1上的容器发出的IP包,基于本地路由表发送给10.100.1.1网关路由器,然后路由器收到IP包查看目的IP,通过本地路由表找到下一跳地址发送到node2,最终到达目的容器。这种方案,我们是可以基于underlay 网络来实现,只要底层支持BGP网络,可以和我们RR节点建立EBGP关系来交换集群内的路由信息。

以上就是kubernetes 常用的几种网络方案了,在公有云场景下一般用云厂商提供的cni plugin即可,大多数情况都是大二层子网内可以使用flannel host-gw这种更简单的方案,对于三层可以使用flannel vxlan 或者calico ipip,而私有物理机房环境中,Calico bgp项目更加适合。根据自己的实际场景,再选择合适的网络方案。

参考

  • https://github.com/coreos/flannel/blob/master/Documentation/backends.md
  • https://coreos.com/flannel/
  • https://docs.projectcalico.org/getting-started/kubernetes/
  • https://www.kancloud.cn/willseecloud/kubernetes-handbook/1321338

监控后传

这些年随着互联网的进步,监控系统也越来越完善,那接下来就谈谈如何理解监控,以及监控要做到什么

最近一直在思考,监控到底该是一个什么样子,为什么要监控?

为什么要监控,我认为监控一个系统有多个原因

  1. 分析长期趋势
    通过对监控样本数据的持续收集和统计,对监控指标进行长期趋势分析
  2. 周期的比较(环比)
    通过跨时间的比较,或者是观察实验组与控制组之间的区别
  3. 报警
    出现故障,需要立刻修复,或者可能很快会出故障,需要提前介入查看
  4. 监控大盘(黄金指标)
    系统的展示服务的一些基本问题,具体参考4个黄金指标
  5. 回溯分析
    经过对长期数据存储分析,并对数据进行多维度的回溯分析,能够帮助我们透视化了解系统的异常情况,主动发现潜在隐患

监控系统的核心本质是什么

大家都知道,无论故障发生的概率有多小,只要有出现的可能,它总会复现。

海恩法则指出:每一起严重事故的背后,必然有29次轻微事故和300期未遂先兆以及1000起事故隐患。

法则强调俩点:
一是事故的发生是量的积累的结果
二是再好的技术,再完美的规章,在实际操作层面,也无法取代人自身的素质和责任心

因此我们需要对自己质疑

  1. 什么东西出故障了(现象)
  2. 为什么出故障(原因)

<<SRE:Goole运维解密>> 一书指出,监控系统需要能够有效的支持白盒监控和黑盒监控

可能有点懵,简单介绍一下 白盒监控、黑盒监控。

白盒监控大量依赖对系统内部信息的监测,如系统日志、抓取提供指标信息的http节点等,因此可以监测到即将发生的问题及哪些重试所掩盖的问题等。

黑盒监控是面向现象的,代表了目前正在发生的,而非预测会发生的问题,即(系统现在有故障)。

监控系统的4个黄金指标(延迟、流量、错误、饱和度)

延迟:服务处理某个请求所需要的时间
流量:使用系统中的某个高层次的指标对系统负载需求所进行的度量
错误:请求失败的速率
饱和度:服务容量有多“满”

如果我们的度量所有这4个黄金指标,同时在某个指标出现故障时发出警报,能做到这些,服务的监控就基本差不多了。

If you are looking for a distinctive solution to obtain what you wish in the male, there can be a sugar daddy distinction that is certainly ending through getting it. Sometimes there exists even more to a sugardaddy than meets the attention in fact it is your choice to determine should you can choose a sugardaddy kind or another. Typically the sugar daddy kind functions the same as virtually any various other sort of sugardaddy and is likely to make anyone looks fine. He will as well supply you with a piece of extra money and because of this extra cash you may enjoy yourself and have absolutely fun with out experiencing that it will strain your money. So what now kind of sugardaddy is perfect for a person? A great way to determine will be to choosing a contractor and even type “what form of sugardaddy is good for me”. Utilizing the web it is possible you just read some good websites to find out which kind of sugar daddy is perfect for a person. You can find websites that will assist you find form of sugar daddy is ideal for you because you never want to squander your time and energy on one of those who will not work effective for you. These websites can confirm each of the great things about by using a gentleman that will give you extra money and everything you have to do is definitely subscribe. There are various web sites similar to this however, you ought not take the former that you just check out over the internet because it may be some sort of junk mail. When using a website that can assist you learn what kind of sugar daddy kind is great for after this you you might probably the most profit. It can be more costly define sugar daddy than you accustomed to having to pay although once you find a web site that is doing work you will realize that it can save you some huge cash. After you commence finding the extra money and even enjoying yourself without having feeling as you usually are damaging your finances you will are aware that an individual built the best selection. You can also discover learning to make by far the most of the sugardaddy by being able to experience the sexual experience once the period will come. From this article you can see we have a whole lot to know about what sort of sugar daddy is designed for a person. Ensure that you are getting what you wish and be happy with the sort of guy that you’ll be with.

If you are looking for a distinctive solution to obtain what you wish in the male, there can be a sugar daddy distinction that is certainly ending through getting it. Sometimes there exists even more to a sugardaddy than meets the attention in fact it is your choice to determine should you can choose a sugardaddy kind or another.

Typically the sugar daddy kind functions the same as virtually any various other sort of sugardaddy and is likely to make anyone looks fine. He will as well supply you with a piece of extra money and because of this extra cash you may enjoy yourself and have absolutely fun with out experiencing that it will strain your money.

So what now kind of sugardaddy is perfect for a person? A great way to determine will be to choosing a contractor and even type “what form of sugardaddy is good for me”.

Utilizing the web it is possible you just read some good websites to find out which kind of sugar daddy is perfect for a person. You can find websites that will assist you find form of sugar daddy is ideal for you because you never want to squander your time and energy on one of those who will not work effective for you.

These websites can confirm each of the great things about by using a gentleman that will give you extra money and everything you have to do is definitely subscribe. There are various web sites similar to this however, you ought not take the former that you just check out over the internet because it may be some sort of junk mail.

When using a website that can assist you learn what kind of sugar daddy kind is great for after this you you might probably the most profit. It can be more costly define sugar daddy than you accustomed to having to pay although once you find a web site that is doing work you will realize that it can save you some huge cash.

After you commence finding the extra money and even enjoying yourself without having feeling as you usually are damaging your finances you will are aware that an individual built the best selection. You can also discover learning to make by far the most of the sugardaddy by being able to experience the sexual experience once the period will come.

From this article you can see we have a whole lot to know about what sort of sugar daddy is designed for a person. Ensure that you are getting what you wish and be happy with the sort of guy that you’ll be with.

Data rooms: let your business be successful | data room

Want more features? Work with partners better and organize projects more proficiently? Do you care about data protection and cell work with all of them? Then you ought to pay attention to . They may be used by normally from the Fortune-500 list, since the mentioned tool allows businesses to act on maximum!

More features and mobility

Virtual data rooms are a cloud storage area that allows a modern business to operate without any obstacles. Firstly, the development allows you to exchange data and work with them in a safe function. You will be able to share commercial and confidential documents with associates and clients around the world. Secondly, the platform permits faster and easier to determine transactions of any complexness, as well as execute examinations and audits. Thirdly, it will be easy and convenient for you to entice a specialist and organize the work of a remote control team.

It might be worth saying that data rooms allow you to quickly and efficiently work with papers. You will spend even less time on routine responsibilities and emphasis more in decision making and management top quality. associated with business even more mobile, deal with more automated and much easier, and all techniques are more secure. Indeed, it really is safety which is most important element of stable creation and powerful functioning.

More Trustworthiness and Security for Data

Data rooms have been built to wonderful web app security standards. This means that solutions and functions are the most efficient. The coders used the most modern strategies of encryption and key storage space, the most relevant anti-virus systems. As a result, the reliability of data rooms corresponds to the consistency of banking systems. Not merely the software is extremely secure, nonetheless also data centers. They may have strict physical access handles, as well as wonderful safety protocols in case of flame, flood or perhaps earthquake. And, of course , current file backups.

Do not forget that one of the most important aspects of control is that you simply. Before sending each file, it is you who will collection the get parameters, added restrictions. Plus the user who all receives the file will need to confirm his identity in many stages and will also be able to job only inside the parameters set for him. It is important to know that you can often cancel use of a file, regardless if it has already been downloaded. Each and every one user activities with files are saved in a wonderful journal, that you will have get.

Extra offers and best service plan

Electronic data rooms are a great way for you to do more, with less attempt. You can make the operation of the company much easier and more rewarding. High-quality technical support will also provide you with this. Ask questions and seek specialist advice whenever you want. Also, if the need takes place, they will be capable to help you with the development of unique functions, digitization and organization of documents. You can always count on the prompt image resolution of concerns and support for your organization.

If you want to know even more about best virtual data room, then simply just start using them. This can be carried out completely free with respect to thirty in the event you activate quality mode. Not only will you learn the info, but likewise check the top quality, simplicity and effectiveness on the software.

If you are looking somebody to places to meet singles near me date, you ought to be looking for an individual who might be good looking, funny and willing to get out and do some dating. To obtain the perfect guy/girl you must be smart about how precisely you way them. You ought not just take a look at them and enquire if they are interested. The best way to start out is by requesting questions that could give you signs as to the personality. This way you can get to discover them. Here are some great questions to ask a guy/girl to start off with. Asking concerns like these might let them know that you might want to know more about them. You must take time to 3 ingredients . their interests and hobbies. This way you can study how they use their spare time. It is easier to talk about activities or observing a movie with them in case you know how that they spend their very own free time. Once you have found the right guy/girl you may move on to buying date for where to find a girlfriend his or her place. The simplest way to do this should be to go to www.elite-brides.com/review/asianmelodies the local recreation area, playground or perhaps a coffee shop. You need to be able to find to start a date there and discover a nice destination to get a chew to eat.

If you are looking somebody to places to meet singles near me date, you ought to be looking for an individual who might be good looking, funny and willing to get out and do some dating. To obtain the perfect guy/girl you must be smart about how precisely you way them. You ought not just take a look at them and enquire if they are interested. The best way to start out is by requesting questions that could give you signs as to the personality. This way you can get to discover them. Here are some great questions to ask a guy/girl to start off with.

Asking concerns like these might let them know that you might want to know more about them. You must take time to 3 ingredients . their interests and hobbies. This way you can study how they use their spare time. It is easier to talk about activities or observing a movie with them in case you know how that they spend their very own free time.

Once you have found the right guy/girl you may move on to buying date for where to find a girlfriend his or her place. The simplest way to do this should be to go to www.elite-brides.com/review/asianmelodies the local recreation area, playground or perhaps a coffee shop. You need to be able to find to start a date there and discover a nice destination to get a chew to eat.