在 Kubernetes 中要实现容器之间网络的隔离，是通过一个专门的 API 对象 NetworkPolicy（网络策略）来实现的，要让网络策略生效，就需要特定的网络插件支持，目前已经实现了 NetworkPolicy 的网络插件包括 Calico、Weave 和 kube-router 等项目，但是并不包括 Flannel 项目。所以说，如果想要在使用 Flannel 的同时还使用 NetworkPolicy 的话，你就需要再额外安装一个网络插件，比如 Calico 项目，来负责执行 NetworkPolicy。本测试环境使用的是 Calico 网络插件,可以直接使用
默认情况下 Pod 是可以接收来自任何发送方的请求，也可以向任何接收方发送请求。如果要对这个情况作出限制，就必须通过 NetworkPolicy 对象来指定。
以下定义了一个网络策略资源清单文件,内容如下：
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: network-policy
namespace: default
spec:
podSelector:
matchLabels:
app: nginx
policyTypes:
- Ingress
- Egress
ingress:
- from:
- ipBlock:
cidr: 10.233.0.0/16
except:
- 10.233.93.0/24
- namespaceSelector:
matchLabels:
project: test
- podSelector:
matchLabels:
app: busybox
- ports:
- protocol: TCP
port: 80
egress:
- to:
- ipBlock:
cidr: 10.233.93.0/24
ports:
- protocol: TCP
port: 5978
Ingress参数解释：
spec:
podSelector:
matchLabels:
app: nginx ###定义了pod对象，表示当前ns中标签为app: nginx的pod
如果想要NetworkPolicy作用于当前ns中的所有pod，则用如下形式：
spec:
podSelector: {}
spec:
policyTypes: ###网络策略的类型
- Ingress
- Egress
每个 NetworkPolicy 包含一个 policyTypes 列表，可以是一个 Ingress、Egress 或者都包含，该字段表示给当前策略是否应用于所匹配的 Pod 的入口流量、出口流量或者二者都包含，如果没有指定 policyTypes，则默认情况下表示 Ingress 入口流量，如果配置了任何出口流量规则，则将指定为 Egress。
ingress: ###配置pod的ingress策略
- from:
- ipBlock: ###配置允许的网络访问
cidr: 10.233.0.0/16
except: ###不允许某个网络访问
- 10.233.93.0/24
- namespaceSelector: ###允许标签为project: test的ns下的所有的pod访问
matchLabels:
project: test
- podSelector: ###此处代表当前ns下的pod，默认是允许同一ns下的pod互通，添加了此处选项代表只允许当前ns下label为app: busybox的pod访问label为app: nginx的pod应用
matchLabels:
app: busybox
- ports: ####允许被访问的端口
- protocol: TCP
port: 80
一旦 Pod 被 NetworkPolicy 选中，那么这个 Pod 就会进入“拒绝所有”（Deny All）的状态，即这个 Pod 既不允许被外界访问，也不允许对外界发起访问，所以 NetworkPolicy 定义的规则，其实就是“白名单”了。
ingress测试
在default的ns下启动两个web服务的pod，如下：
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 9d 10.233.96.3 node2 <none> <none>
web 1/1 Running 0 9d 10.233.96.4 node2 <none> <none>
创建networkpolicy，如下：
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: denynetwork
spec:
podSelector:
matchLabels:
app: nginx ####被访问pod的label标签
policyTypes:
- Egress
- Ingress ###添加规则类型
ingress:
- from:
- podSelector: ###此处代表当前ns下的pod，默认是允许同一ns下的pod互通，添加了此处选项代表只允许当前ns下label为app: web的pod访问label为app: nginx的pod应用
matchLabels:
app: web
创建如下：
[root@master ~]# kubectl get networkpolicy
NAME POD-SELECTOR AGE
denynetwork app=nginx 9d ###可以看到被访问的pod标签
[root@master ~]#
podSelector测试：
##############
如上，web pod位于node2上，进入pod的网络命令空间
[root@node2 ~]# crictl ps | grep web
dcf892535ed93 3f8a00f137a0d 9 days ago Running count 0 205811ac7b0f0 web
[root@node2 ~]#
[root@node2 ~]# crictl inspect dcf892535ed93 | grep -i pid
"pid": 16814,
"pid": 1
"type": "pid"
[root@node2 ~]#
[root@node2 ~]# nsenter -t 16814 -n bash ####进入pod的网络命令空间
[root@node2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether 9a:6b:07:ab:e4:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.233.96.4/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::986b:7ff:feab:e40d/64 scope link
valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.3 #####此处可以看到无放访问nginx pod
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 5999ms
[root@node2 ~]#
#######################
######################
为web pod添加app: web的标签，如下：
root@master ~]# kubectl label po web app=web
pod/web labeled
[root@master ~]#
#####################
再次测试访问nginx，如下：
root@master ~]# ssh node2
Last login: Sat Mar 4 14:26:28 2023 from 192.168.5.240
[root@node2 ~]# nsenter -t 16814 -n bash
[root@node2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether 9a:6b:07:ab:e4:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.233.96.4/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::986b:7ff:feab:e40d/64 scope link
valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=63 time=0.333 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=63 time=0.127 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.127/0.230/0.333/0.103 ms
[root@node2 ~]# curl 10.233.96.3 ####可以正常访问服务
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node2 ~]#
namespaceSelector测试
编辑networkpolicy.yaml文件
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: denynetwork
spec:
podSelector:
matchLabels:
app: nginx ####被访问pod的label标签
policyTypes:
- Egress
- Ingress ###添加规则类型
ingress:
- from:
- namespaceSelector: ###添加了此处选项代表只允许label为prject:test的ns下的pod访问label为app: nginx的pod应用
matchLabels:
project: test
创建如下：
[root@master ~]# kubectl get networkpolicy
NAME POD-SELECTOR AGE
denynetwork app=nginx 9d ###可以看到被访问的pod标签
创建ns以及测试pod
#######创建ns
[root@master ~]# kubectl create ns test
#####创建测试pod
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: test #####选择test的ns
spec:
containers:
- name: busybox
image: docker.io/library/busybox:latest
imagePullPolicy: IfNotPresent
command: ["init"]
启动pod如下：
[root@master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 9d 10.233.90.4 node1 <none> <none>
[root@master ~]#
###################
进入busybox的网络命令，如下：
[root@node1 ~]# crictl ps | grep busybox
755e001322ef6 2fb6fc2d97e10 9 days ago Running busybox 0 b5e1dfe3d3746 busybox
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# crictl inspect 755e001322ef6| grep -i pid
"pid": 44236,
"pid": 1
"type": "pid"
[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether 42:2b:78:3a:6a:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.233.90.4/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::402b:78ff:fe3a:6ac1/64 scope link
valid_lft forever preferred_lft forever
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms
[root@node1 ~]#
##############
为test ns添加标签，如下：
[root@master ~]# kubectl label ns test project=test
namespace/test labeled
[root@master ~]#
再次测试如下：
[root@node1 ~]#
[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=62 time=2.35 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=62 time=1.46 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.468/1.910/2.352/0.442 ms
[root@node1 ~]# curl 10.233.96.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node1 ~]#
ipBlock测试
编辑networkpolicy.yaml文件：
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: denynetwork
namespace: default
spec:
podSelector:
matchLabels:
app: nginx
policyTypes:
- Ingress
- Egress
ingress:
- from:
- ipBlock:
cidr: 10.233.0.0/16 #####允许访问的网段
except:
- 10.233.93.0/24 ####排除此网段（此网段位于node1，所以测试pod也位于node1）
创建测试pod
###########创建pod，如下：
[root@master ~]# kubectl get po -o wide -n test
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 9d 10.233.90.4 node1 <none> <none>
##########
进入pod的网络命名空间
[root@master ~]# ssh node1
Last login: Sat Mar 4 14:55:41 2023 from 192.168.5.240
[root@node1 ~]# crictl ps | grep busybox
755e001322ef6 2fb6fc2d97e10 9 days ago Running busybox 0 b5e1dfe3d3746 busybox
[root@node1 ~]#
[root@node1 ~]# crictl inspect 755e001322ef6 | grep -i pid
"pid": 44236,
"pid": 1
"type": "pid"
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# nsenter -t 44236 -n bash
[root@node1 ~]#
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
^C
--- 10.233.96.3 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms
[root@node1 ~]#
#############
修改networkpolicy，取消expect参数
[root@master ~]# kubectl edit networkpolicy denynetwork
networkpolicy.networking.k8s.io/denynetwork edited
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:
PodSelector: app=nginx
Allowing ingress traffic:
To Port: <any> (traffic allowed to all ports)
From:
IPBlock:
CIDR: 10.233.0.0/16
Except: ####已经去掉了10.233.93.0/24的网段的限制
Allowing egress traffic:
<none> (Selected pods are isolated for egress connectivity)
Policy Types: Egress, Ingress
[root@master ~]#
#######################
再此测试如下：
[root@node1 ~]#
[root@node1 ~]# ping 10.233.96.3
PING 10.233.96.3 (10.233.96.3) 56(84) bytes of data.
64 bytes from 10.233.96.3: icmp_seq=1 ttl=62 time=2.76 ms
64 bytes from 10.233.96.3: icmp_seq=2 ttl=62 time=1.55 ms
^C
--- 10.233.96.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.556/2.158/2.761/0.604 ms
[root@node1 ~]# curl 10.233.96.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node1 ~]#
egress参数解释
每个 NetworkPolicy 包含一个 egress 规则的白名单列表。每个规则都允许匹配 to 和 port 部分的流量。比如我们这里示例规则的配置：
egress:
- to:
- ipBlock:
cidr: 10.0.0.0/24 ##允许访问10.0.0.0段的80端口
ports:
- protocol: TCP
port: 80
表示 Kubernetes 会拒绝被隔离 Pod 对外发起任何请求，除非请求的目的地址属于 10.0.0.0/24 网段，并且访问的是该网段地址的 80 端口。
创建networkpolicy测试：
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: denynetwork
namespace: default
spec:
podSelector:
matchLabels:
app: nginx
policyTypes:
- Ingress
- Egress
egress:
- to:
- ipBlock:
cidr: 10.233.90.0/24 #####允许访问的网段
ports:
- protocol: TCP
port: 80 ###允许被访问网络的端口
#######创建策略查看
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:
PodSelector: app=nginx
Allowing ingress traffic:
<none> (Selected pods are isolated for ingress connectivity)
Allowing egress traffic:
To Port: 80/TCP
To:
IPBlock:
CIDR: 10.233.90.0/24
Except:
Policy Types: Ingress, Egress
[root@master ~]#
###########找到app=nginx的pod并访问
[root@master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
egress-test 1/1 Running 0 70s 10.233.90.6 node1 <none> <none>
[root@master ~]# kubectl get po -o wide | grep nginx
nginx 1/1 Running 0 9d 10.233.96.3 node2 <none> <none>
###############进入app=nginx的pod的命令空间访问
[root@node2 ~]# crictl ps | grep nginx
27fd22bdf596d 3f8a00f137a0d 9 days ago Running count 0 cb2afd0d86bcf nginx
[root@node2 ~]# crictl inspect 27fd22bdf596d| grep -i pid
"pid": 11243,
"pid": 1
"type": "pid"
[root@node2 ~]# nsenter -t 11243 -n bash
[root@node2 ~]#
[root@node2 ~]# ping 10.233.90.6 ###因为只允许了tcp协议，所以ping不通
PING 10.233.90.6 (10.233.90.6) 56(84) bytes of data.
^C
--- 10.233.90.6 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 2999ms
[root@node2 ~]# curl 10.233.90.6 ####如下可正常访问80端口
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node2 ~]#
###########编辑networkpolicy ，将80端口改为81
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:
PodSelector: app=nginx
Not affecting ingress traffic
Allowing egress traffic:
To Port: 81/TCP
To:
IPBlock:
CIDR: 10.233.90.0/24
Except:
Policy Types: Egress
[root@master ~]#
#############再次访问10.233.90.6的80端口如下：
[root@node2 ~]# curl 10.233.90.6 ###无法访问
^C
[root@node2 ~]#
###########编辑networkpolicy，放通整个网段测试
[root@master ~]# kubectl edit networkpolicy denynetwork
networkpolicy.networking.k8s.io/denynetwork edited
[root@master ~]# kubectl describe networkpolicy denynetwork
Name: denynetwork
Namespace: default
Created on: 2023-02-22 22:39:54 +0800 CST
Labels: <none>
Annotations: <none>
Spec:
PodSelector: app=nginx
Not affecting ingress traffic
Allowing egress traffic:
To Port: <any> (traffic allowed to all ports)
To:
IPBlock:
CIDR: 10.233.90.0/24
Except:
Policy Types: Egress
[root@master ~]#
########再次访问10.233.90.6，如下：
[root@node2 ~]# ping 10.233.90.6
PING 10.233.90.6 (10.233.90.6) 56(84) bytes of data.
64 bytes from 10.233.90.6: icmp_seq=1 ttl=62 time=1.56 ms
^C
--- 10.233.90.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.560/1.560/1.560/0.000 ms
[root@node2 ~]#
[root@node2 ~]# curl 10.233.90.6
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@node2 ~]#
########################
但是无法访问其他节点的pod，即使是同网段和节点，如下：
[root@master ~]# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 9d 10.233.96.3 node2 <none> <none>
web 1/1 Running 0 9d 10.233.96.4 node2 <none> <none>
[root@node2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether 56:52:52:7c:82:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.233.96.3/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5452:52ff:fe7c:8281/64 scope link
valid_lft forever preferred_lft forever
[root@node2 ~]# ping 10.233.96.4
PING 10.233.96.4 (10.233.96.4) 56(84) bytes of data.
^C
--- 10.233.96.4 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms
[root@node2 ~]# curl 10.233.96.4
^C
[root@node2 ~]#