Kubernetes网络通信： (1) 容器间通信：同一个Pod内的多个容器间的通信, lo (2) Pod通信：Pod IP <--> Pod IP (3) Pod与Service通信：PodIP <--> ClusterIP (4) Service与集群外部客户端的通信；
CNI: flannel calico canel kube-router ... 解决方案： 虚拟网桥 多路复用：MacVLAN 硬件交换：SR-IOV [root@master ~]# cat /etc/cni/net.d/10-flannel.conflist { "name": "cbr0", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] }
flannel： 支持多种后端： VxLAN (1) vxlan (2) Directrouting host-gw: Host Gateway #不推荐，只能在二层网络中，不支持跨网络，如果有成千上万的Pod，容易产生广播风暴 UDP: 性能差 flannel的配置参数： Network：flannel使用的CIDR格式的网络地址，用于为Pod配置网络功能； 10.244.0.0/16 -> master: 10.244.0.0/24 node01: 10.244.1.0/24 ... node255: 10.244.255.0./24 10.0.0.0/8 10.0.0.0/24 ... 10.255.255.0/24 SubnetLen：把Network切分子网供各节点使用时，使用多长的掩码进行切分，默认为24位； SubnetMin：10.244.10.0/24 SubnetMax: 10.244.100.0/24 Backend：vxlan, host-gw, udp vxlan:
flannel Pod间跨主机通信原理抓包[root@master ~]# ip rdefault via 172.20.0.1 dev ens3 proto dhcp metric 100
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1 10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.20.0.0/16 dev ens3 proto kernel scope link src 172.20.0.91 metric 100
从路由可以看出 Pod cni0 -----> flannel.1 -----> 物理网卡 -----> 对端 [root@master ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:62:b2:ca brd ff:ff:ff:ff:ff:ff inet 172.20.0.91/16 brd 172.20.255.255 scope global noprefixroute dynamic ens3 valid_lft 2412sec preferred_lft 2412sec 3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:40:09:09:b4 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether 26:e1:60:30:27:c1 brd ff:ff:ff:ff:ff:ff inet 10.244.0.0/32 scope global flannel.1 valid_lft forever preferred_lft forever 5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff inet 10.244.0.1/24 scope global cni0 valid_lft forever preferred_lft forever 6: veth0f580b07@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default link/ether a2:98:01:9b:b5:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 7: vethb8510761@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default link/ether 1a:00:6b:77:33:fc brd ff:ff:ff:ff:ff:ff link-netnsid 1 113703: vethfc114a8b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default link/ether fa:ec:f9:ee:42:93 brd ff:ff:ff:ff:ff:ff link-netnsid 3 30029: veth023640f@if30028: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether d2:8a:82:e6:8f:40 brd ff:ff:ff:ff:ff:ff link-netnsid 5 13217: br-973161700d44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:54:2e:ec:14 brd ff:ff:ff:ff:ff:ff inet 172.18.0.1/16 brd 172.18.255.255 scope global br-973161700d44 valid_lft forever preferred_lft forever 13227: vethfdee34f@if13226: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default link/ether 7e:3e:6a:e2:44:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 2 13229: veth49aa01b@if13228: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-973161700d44 state UP group default link/ether b2:27:38:6e:4d:da brd ff:ff:ff:ff:ff:ff link-netnsid 4
cni0 提供Pod网络共享的地址来源
flannel.1 是对数据包进行 vxlan封装
[root@master ~]# brctl show cni0 bridge name bridge id STP enabled interfaces cni0 8000.0a580af40001 no veth0f580b07 vethb8510761 vethfc114a8b [root@master ~]# brctl show flannel.1 bridge name bridge id STP enabled interfaces flannel.1 can't get info Operation not supported
抓包
15:13:26.796845 IP 172.20.0.93.52954 > 172.20.0.76.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.1.175 > 10.244.2.223: ICMP echo request, id 3072, seq 221, length 64