k8s flannel 网络抓包实践

初始环境如下，对两台机器的 docker0、flannel0、veth设备对进行了tcpdump抓包。

从busybox 发起ping  nginx pod。

k8s-node-1节点信息

[root@k8s-node-1 ~]# bridge fdb
01:00:5e:00:00:01 dev enp0s3 self permanent
33:33:00:00:00:01 dev enp0s3 self permanent
33:33:ff:84:23:95 dev enp0s3 self permanent
33:33:ff:23:4d:1b dev enp0s3 self permanent
33:33:00:00:00:01 dev docker0 self permanent
01:00:5e:00:00:01 dev docker0 self permanent
33:33:ff:f4:d9:e3 dev docker0 self permanent
02:42:85:f4:d9:e3 dev docker0 master docker0 permanent
02:42:85:f4:d9:e3 dev docker0 vlan 1 master docker0 permanent
3a:1c:f3:ca:0b:24 dev veth5655be9 vlan 1 master docker0 permanent
3a:1c:f3:ca:0b:24 dev veth5655be9 master docker0 permanent
33:33:00:00:00:01 dev veth5655be9 self permanent
01:00:5e:00:00:01 dev veth5655be9 self permanent
33:33:ff:ca:0b:24 dev veth5655be9 self permanent
[root@k8s-node-1 ~]# ip r
default via 192.168.2.1 dev enp0s3 proto static metric 100 
10.0.0.0/16 dev flannel0 proto kernel scope link src 10.0.63.0 
10.0.63.0/24 dev docker0 proto kernel scope link src 10.0.63.1 
192.168.2.0/24 dev enp0s3 proto kernel scope link src 192.168.2.102 metric 100 
[root@k8s-node-1 ~]# ip neigh show
192.168.2.101 dev enp0s3 lladdr 08:00:27:d3:74:be REACHABLE
192.168.2.130 dev enp0s3 lladdr 98:3b:8f:ac:44:cd DELAY
10.0.63.2 dev docker0 lladdr 02:42:0a:00:3f:02 STALE
192.168.2.103 dev enp0s3 lladdr 08:00:27:f0:7e:1a STALE
10.0.63.4 dev docker0 lladdr 02:42:0a:00:3f:04 STALE
10.0.63.3 dev docker0 lladdr 02:42:0a:00:3f:03 STALE
192.168.2.1 dev enp0s3 lladdr 2c:b2:1a:4c:09:09 STALE
192.168.2.104 dev enp0s3  FAILED

k8s-node-2节点信息

[root@k8s-node-2 ~]# bridge fdb
01:00:5e:00:00:01 dev enp0s3 self permanent
33:33:00:00:00:01 dev enp0s3 self permanent
33:33:ff:84:23:95 dev enp0s3 self permanent
33:33:ff:23:4d:1b dev enp0s3 self permanent
33:33:ff:63:6c:6b dev enp0s3 self permanent
33:33:00:00:00:01 dev docker0 self permanent
01:00:5e:00:00:01 dev docker0 self permanent
33:33:ff:61:e1:9a dev docker0 self permanent
02:42:78:61:e1:9a dev docker0 master docker0 permanent
02:42:78:61:e1:9a dev docker0 vlan 1 master docker0 permanent
86:90:5f:c4:21:54 dev vethd946e9a master docker0 permanent
86:90:5f:c4:21:54 dev vethd946e9a vlan 1 master docker0 permanent
33:33:00:00:00:01 dev vethd946e9a self permanent
01:00:5e:00:00:01 dev vethd946e9a self permanent
33:33:ff:c4:21:54 dev vethd946e9a self permanent
[root@k8s-node-2 ~]# ip r
default via 192.168.2.1 dev enp0s3 proto static metric 100 
10.0.0.0/16 dev flannel0 proto kernel scope link src 10.0.14.0 
10.0.14.0/24 dev docker0 proto kernel scope link src 10.0.14.1 
192.168.2.0/24 dev enp0s3 proto kernel scope link src 192.168.2.103 metric 100 
[root@k8s-node-2 ~]# ip neigh show
192.168.2.102 dev enp0s3 lladdr 08:00:27:9a:7b:2e STALE
10.0.14.4 dev docker0  FAILED
192.168.2.101 dev enp0s3 lladdr 08:00:27:d3:74:be REACHABLE
10.0.14.3 dev docker0  FAILED
192.168.2.130 dev enp0s3 lladdr 98:3b:8f:ac:44:cd DELAY
192.168.2.1 dev enp0s3 lladdr 2c:b2:1a:4c:09:09 STALE
10.0.14.2 dev docker0 lladdr 02:42:0a:00:0e:02 STALE
[root@k8s-node-2 ~]# 

日志如下，以下的按照抓包的时间先后列出日志：

1、发起ping 的服务器：192.168.2.103

veth

Last login: Sun Mar 22 15:13:25 2020 from 192.168.2.130
[root@k8s-node-2 ~]# tcpdump -i vethd946e9a
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vethd946e9a, link-type EN10MB (Ethernet), capture size 262144 bytes
15:14:20.299074 ARP, Request who-has k8s-node-2 tell 10.0.14.2, length 28
15:14:20.299080 ARP, Request who-has k8s-node-2 tell 10.0.14.2, length 28
15:14:20.299087 ARP, Reply k8s-node-2 is-at 02:42:78:61:e1:9a (oui Unknown), length 28
15:14:20.299101 IP 10.0.14.2 > 10.0.63.2: ICMP echo request, id 55808, seq 0, length 64
15:14:20.300401 IP 10.0.63.2 > 10.0.14.2: ICMP echo reply, id 55808, seq 0, length 64
15:14:25.301493 ARP, Request who-has 10.0.14.2 tell k8s-node-2, length 28
15:14:25.301530 ARP, Reply 10.0.14.2 is-at 02:42:0a:00:0e:02 (oui Unknown), length 28

docker0

Last login: Sun Mar 22 14:44:15 2020 from 192.168.2.130
[root@k8s-node-2 ~]# tcpdump -i docker0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:14:20.299081 ARP, Request who-has k8s-node-2 tell 10.0.14.2, length 28
15:14:20.299086 ARP, Reply k8s-node-2 is-at 02:42:78:61:e1:9a (oui Unknown), length 28
15:14:20.299107 IP 10.0.14.2 > 10.0.63.2: ICMP echo request, id 55808, seq 0, length 64
15:14:20.300398 IP 10.0.63.2 > 10.0.14.2: ICMP echo reply, id 55808, seq 0, length 64
15:14:25.301488 ARP, Request who-has 10.0.14.2 tell k8s-node-2, length 28
15:14:25.301532 ARP, Reply 10.0.14.2 is-at 02:42:0a:00:0e:02 (oui Unknown), length 28

flannel0

[root@k8s-node-2 ~]# tcpdump -i flannel0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on flannel0, link-type RAW (Raw IP), capture size 262144 bytes
15:14:20.299113 IP k8s-node-2 > 10.0.63.2: ICMP echo request, id 55808, seq 0, length 64
15:14:20.300392 IP 10.0.63.2 > k8s-node-2: ICMP echo reply, id 55808, seq 0, length 64

2、响应ping 的服务器：192.168.2.102

flannel0

Last login: Sun Mar 22 15:10:46 2020 from 192.168.2.130
[root@k8s-node-1 ~]# tcpdump -i flannel0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on flannel0, link-type RAW (Raw IP), capture size 262144 bytes
15:14:20.472666 IP 10.0.14.0 > 10.0.63.2: ICMP echo request, id 55808, seq 0, length 64
15:14:20.472719 IP 10.0.63.2 > 10.0.14.0: ICMP echo reply, id 55808, seq 0, length 64

docker0

[root@k8s-node-1 ~]# tcpdump -i docker0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:14:20.472681 IP 10.0.14.0 > 10.0.63.2: ICMP echo request, id 55808, seq 0, length 64
15:14:20.472707 ARP, Request who-has k8s-node-1 tell 10.0.63.2, length 28
15:14:20.472711 ARP, Reply k8s-node-1 is-at 02:42:85:f4:d9:e3 (oui Unknown), length 28
15:14:20.472717 IP 10.0.63.2 > 10.0.14.0: ICMP echo reply, id 55808, seq 0, length 64

veth

Last login: Sun Mar 22 15:11:23 2020 from 192.168.2.130
[root@k8s-node-1 ~]# tcpdump -i veth5655be9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth5655be9, link-type EN10MB (Ethernet), capture size 262144 bytes
15:14:20.472686 IP 10.0.14.0 > 10.0.63.2: ICMP echo request, id 55808, seq 0, length 64
15:14:20.472704 ARP, Request who-has k8s-node-1 tell 10.0.63.2, length 28
15:14:20.472706 ARP, Request who-has k8s-node-1 tell 10.0.63.2, length 28
15:14:20.472712 ARP, Reply k8s-node-1 is-at 02:42:85:f4:d9:e3 (oui Unknown), length 28
15:14:20.472715 IP 10.0.63.2 > 10.0.14.0: ICMP echo reply, id 55808, seq 0, length 64

2、结论如下：