SpiderPool is derived from the accumulated experience of container network landing practice. It is an open source native container network IPAM plug-in ( github: https://github.com/spidernet-io/spiderpool) of "Daocloud Daoke", which is mainly used to implement containers with Underlay CNI The cloud platform finely manages and allocates IP.
01
Background of the birth of SpriderPool
Cloud nativeization of traditional applications
First of all, in some traditional industries, such as manufacturing, education, energy and other enterprises, traditional applications are usually directly uploaded to the cloud. These applications have not been transformed by microservices, and still need to be accessed through a certain fixed IP, and strong management and control of such IP is required, such as strict firewall control. Therefore, more flexible and efficient IP management capabilities are required to meet the requirements of such scenarios.
Secondly, in the process of traditional application micro-services, users will gradually containerize some applications, and some containerized applications need to provide external access IPs to achieve interconnection inside and outside the cluster. Therefore, such applications also need to use fixed and externally accessible IP addresses . IP.
In addition, based on the strong management and control requirements for external access IP, in the customer's implementation practice, the IP allocated through the Underlay CNI needs to be connected to the traditional physical subnet planning to directly provide external services, and considering the security of the container network, it needs to be installed in the firewall. Configure and release the used IP, and the overall IP management process and specifications are lengthy and complicated . Moreover, the IP resources in the Underlay CNI are relatively scarce, so it is necessary to carry out strong allocation and control of the Underlay IP . When the IP address release and allocation fail, it needs to be recycled in time to prevent waste of resources.
It can be seen that the cloud-native container network faces the following challenges:
There is a strong demand for fixed and external IP, and a strong demand for flexible and efficient IPAM mechanisms.
02
Underlay CNI
vs Overlay CNI 特性
Before introducing the SpiderPool IPAM plug-in, let's compare the differences between Underlay CNI and Overlay CNI.
Underlay CNI IP Management Features
1. The Pod is connected to the physical network planning, without NAT conversion, and directly communicates with the outside of the cluster;
2. The IP used needs to be opened through the firewall, requires strong control, and requires a fixed Pod IP;
3. The IP resources of the Underlay CNI are limited, so the problem of IP conflicts inside and outside the cluster is highlighted. When the IP cannot be recycled in time, the new Pod will fail to start.
Overlay CNI IP Management Features
1. The Pod Subnet is independently planned, and the external communication between the Pod and the cluster needs to be processed by NAT. Only the node IP is exposed to the outside, and the Pod IP does not need to be exposed to the outside;
2. The number of Overlay CNI IP addresses is sufficient, and each Node independently plans CIDR Block;
3. The firewall cannot see the Pod IP, so there is no security risk when providing access to the outside world, and there is no need to open the firewall for strong IP resource management and control. Because of this background, there is no fixed requirement for the Pod IP;
4. The allocation of Overlay IP addresses can ignore the conflicts in the Underlay network, and the problem of reclaiming leaked IPs has little impact.
It can be seen from the comparison that Overlay CNI has weak requirements for IP management and abundant resources. When an IP conflict occurs in the cluster, it will not cause serious consequences.
In the Underlay CNI network solution, in the CNI solution lacking IPAM or weak IP management capabilities (such as SRI-OV, Macvlan, IPVLAN, OVS-CNI), when faced with the above-mentioned complex scenarios such as strong IP management and IP conflict management Then there is nothing you can do.
SpiderPool is an open-source CNI IPAM project dedicated to meeting the needs of the above scenarios . It combines SpiderPool with Underlay CNI to create a cloud-native network solution that can be produced on the cloud. The latest DCE 5.0 launched by "DaoCloud" has commercialized SpiderPool , as the IPAM solution of Underlay CNI (Macvlan, SRI-OV, etc.).
03
SpriderPool
function
In the above background, SpiderPool is mainly designed based on the following issues:
1. Combining with Underlay CNI to provide external access IP, to meet the application load and provide external access capabilities;
2. Provide strong IP management and control capabilities, including subnet management, IP management, and physical layer firewall security management and control;
3. IP fixed capability, when the business application is restarted, the IP assigned to the business application will still be reserved;
4. Anti-IP conflict (enhancement of IP allocation and IP recovery mechanism).
SpiderPool feature list
Features
Node Affinity for IP Pool
Scenes
On different Nodes, the available Subnets may be different, for example:
1. In the same data center, the Nodes connected to the cluster belong to different Subnets;
2. In a single cluster, Node spans different data centers.
In the above scenario, when different copies of the same application are scheduled to different Nodes, different Subnet Underlay IP addresses need to be allocated.
plan
In IP Pool CR, there is nodeAffinity field, which is used to set Node Label Selector to realize IP Pool.
Affinity with Node, when a Pod is scheduled to a Node, the IPAM plug-in can allocate IP addresses from the affinity IP Pool.
Example of creating an IP Pool:
apiVersion: spiderpool.spidernet.io/v1kind: SpiderIPPoolmetadata:name: master-ipv4-ippoolspec:ipVersion: 4subnet: 172.18.41.0/24ips:- 172.18.41.40-172.18.41.42nodeAffinity: #创建 IP Pool 时，添加节点亲和性matchExpressions:- {key: node-role.kubernetes.io/beijing, operator: Exists}
Application Affinity of IP Pool
Scenes
In a strongly regulated network scenario, all Pods can be assigned IP addresses in a fixed IP address range in rotation to match the release policy of the firewall.
plan
Because the IP collection of SpiderPool IP Pool can be large or small, combined with setting the podAffinity of IP Pool, the affinity binding of the same group or multiple groups of applications can be realized, which not only ensures the unity of IP management methods, but also decouples "application expansion" and "IP address expansion" also fixes the range of IP used by the application.
example
apiVersion: spiderpool.spidernet.io/v1kind: SpiderIPPoolmetadata:name: master-ipv4-ippoolspec:ipVersion: 4subnet: 172.18.41.0/24ips:- 172.18.41.40-172.18.41.42podAffinity: # 创建 IP Pool 时增加 Pod 亲和性matchExpressions:- {key: app-dao2048, operator: Exists}
Namespace Affinity for IP Pools
Scenes
An IP Pool can achieve affinity for applications under the same or multiple Namespaces, and reject applications with irrelevant NSs.
plan
By setting the namespaceAffinity of the IP Pool, the affinity of the same or multiple tenants can be achieved, so that the applications that meet the conditions can be allocated IP addresses from the IP Pool.
example
apiVersion: spiderpool.spidernet.io/v1kind: SpiderIPPoolmetadata:name: master-ipv4-ippoolspec:ipVersion: 4subnet: 172.18.41.0/24ips:- 172.18.41.40-172.18.41.42namespaceAffinity: # 创建 SpiderPool 时增加 Namespace 亲和性matchExpressions:- {key: ns-spider, operator: Exists}
Multiple IP Pool backup
Scenes
When the IP addresses in the IP Pool are all allocated and the IPs in the subnet are all used up, related applications will fail when expanding capacity, so multiple standby IP pools can be set up.
plan
Solution 1: Expand more IP address resources to the existing IP Pool.
Solution 2: Bind multiple IP Pools to the application. If the current IP Pool runs out of IP addresses, the IPAM plug-in will try to allocate IP addresses from the rear IP Pool.
example
annotations:ipam.spidernet.io/ippool: |-{"interface": "eth0","ipv4": ["default-ipv4-ippool", "backup-ipv4-ippool"] #创建 工作负载时，指定多个 IP Pool}
Automatic allocation of fixed IP Pool
Scenes
The infrastructure administrator only wants to provide available IP addresses, and does not want to worry about creating an IP Pool with a "fixed IP range"; while the application administrator does not care about the value of the IP address, and only hopes to obtain a specified number of IP addresses. Therefore, the division of responsibilities causes both parties to only want to do the simplest cluster operations. Therefore, the creation of the "fixed IP range" IP Pool has become a workload that neither party wants to undertake.
In addition, when a workload uses up all the fixed IPs and performs rolling release, the new Pod will fail to start because there are no remaining IPs.
plan
At present, Spiderpool can automatically and dynamically create IP Pools for applications, bind IP Pools, and reclaim IP Pools, which not only solves the decoupling of the responsibilities of personnel in the two departments, but also simplifies the operation of "fixed IP range" applications, greatly reducing The cost of managing IP addresses.
In addition, for the scenario where the rolling release fails due to the exhaustion of fixed IPs, Spiderpool provides the number of elastic IPs based on the fixed IP function. When creating workloads (limited to Deployment, ReplicaSet, DaemonSet, StatefulSet, Job, CronJob), its PodTemplateSpec. Specify a certain number of elastic IPs in the annotation to ensure the normal release of the application.
example
annotations:ipam.spidernet.io/subnet: '{"ipv4": ["subnet-demo-v4"], "ipv6": ["subnet-demo-v6"]}'ipam.spidernet.io/ippool-ip-number: "+2" # 当为 “+2”时为弹性 IP ，当 输入为“2”时为指定固定 IP 数量ipam.spidernet.io/ippool-reclaim: "true"
Statefulset Static IP
Scenes
Deployment and Statefulset controllers have different requirements for fixed IP addresses:
1. For Statefulset, the Pod name needs to remain the same before and after restarting the Pod replica, but the Pod UUID is changed and it is stateful. The application administrator hopes that the Pod will be assigned the same IP before and after restarting.
2. For a Deployment, the Pod name and Pod UUID have changed before and after the Pod copy is restarted, so it is stateless. Therefore, it is not necessary for the old and new Pods to use the same IP. We may only want all the copies in the Deployment to use the same IP. The IP is fixed within a certain IP range.
plan
Spiderpool will ensure that the Statefulset Pod continues to obtain the same IP address in restart and rebuild scenarios.
Note: In the process of changing Statefulset from "shrinkage" to "expansion", it is not guaranteed that the newly expanded Pod can obtain the IP address of the previously scaled down Pod.
Example of creating an IP Pool:
apiVersion: apps/v1kind: StatefulSetmetadata:name: demo-stsnamespace: defaultspec:serviceName: "demo-sts-svc"replicas: 1selector:matchLabels:app: demo-ststemplate:metadata:labels:app: demo-stsspec:containers:- image: busyboximagePullPolicy: IfNotPresentname: demo-stscommand: ["/bin/sh", "-c", "trap : TERM INT; sleep infinity & wait"]
Pod with multiple NICs
Scenes
Through the Multus project, multiple CNI network cards can be inserted into the Pod, and different network cards need to be assigned to use different IP Pools, so as to allocate IP addresses under different subnets, and can flexibly specify the default route of the interface.
plan
It can be combined with the NetworkAttachmentDefinition configuration of Multus, and multiple sets of CNI configurations can be configured with different IP Pools.
kind: NetworkAttachmentDefinitionspec:config: |'{ "ipam": { "default_ipv4_ippool": ["ippool1"], "type": "spiderpool" } .... }'
It can be combined with Pod Annotaiton to assign IP addresses in different IP Pools to each network card.
ipam.spidernet.io/ippools: |-[{"interface": "eth0", "ipv4": ["v4-ippool1"], "ipv6": ["v6-ippool1"], "cleangateway": true},{"interface": "eth1", "ipv4": ["v4-ippool2"], "ipv6": ["v6-ippool2"], "cleangateway": false}]
IP Leak Avoidance
challenge
In practice, it is found that if Macvlan and other CNIs are directly used, in some high-concurrency scenarios, it is very easy for the network namespace created by Docker for the container to not be correctly recycled by the kernel after the container dies, and cause IP data confusion. , IP reclassification, misclassification, conflicts and other abnormal phenomena. However, CNIs such as Macvlan have relatively weak IPAM recycling capabilities, resulting in service Pod IP conflicts and resulting in inaccessibility of service applications. In addition, it is found that the problem of network namespace leakage is difficult to locate and reproduce, and the reasons for the occurrence are diverse and cannot be solved from the root cause.
In scenarios such as Pod failure, restart, and reconstruction, whether the IP address of the old Pod copy is recycled to the IP Pool in time will affect the success or failure of the new Pod copy startup. From the perspective of the IPAM component, IP resources are occupied by some "zombie Pods", resulting in a decrease in available IPs.
In the Overlay IPAM scenario, because the CIDR range is large, this problem is not prominent. In the Underlay scenario, IP resources are limited, and some applications require fixed IP address ranges. This problem will affect the healthy operation of the application.
plan
SpiderPool is an open source CNI IPAM project dedicated to meeting the above scenario requirements and avoiding the above IP conflicts. Combining SpiderPool with Underlay CNI, the event-based and periodic dual recovery mechanism makes the "wasted" IP resources the first Time is released in IP Pool.
reserved IP
Scenes
When it is clear that an IP address has been used outside the cluster, in order to avoid IP conflicts, it may be a time-consuming and labor-intensive task to find and remove the IP address from the existing IP Pool instances. Moreover, the network administrator hopes that this IP address will not be allocated in the stock or all future IP Pool resources.
plan
You can set the IP address that you want the cluster not to use in the ReservedIP CR. Therefore, even if the IP address is included in the IP Pool object, the IPAM plug-in will not assign these IP addresses to the Pod. The IP address in ReservedIP can be:
1. Make it clear that the IP address is used by hosts outside the cluster;
2. It is clear that the IP address cannot be used for network communication, such as subnet IP, broadcast IP, etc.
example
apiVersion: spiderpool.spidernet.io/v1kind: SpiderReservedIPmetadata:name: test-ipv4-reservedipspec:ipVersion: 4ips: # 预留 IP 段- 172.18.42.41-172.18.42.49
custom routing
There are many ways to apply and customize routing. Priority from low to high (application routing --> IP Pool). In IP Pool CR, you can specify:
1. Default route and subnet route
kind: spiderippoolspec:gateway: 172.16.0.1routes: [ {“dst”: “192.168.0.0/16”, “gw”: “172.16.0.2”} ]
2. Application level routing:
#在 Pod Annotaiton 中ipam.spidernet.io/routes : [{“dst”: “192.168.0.0/16”, “gw”: “172.16.0.2”}]
04
SpriderPool
CR design
SpiderPool uses CRD (CustomResourceDefinitions) to define resources such as subnets and IP Pools. Through the two-layer storage design of Spider Subnet and Spider IP Pool CRD, IP Pool CR defines IP address sets. Different IP addresses in a Subnet are stored separately. In different IP Pool instances, an IP Pool set can be expanded or reduced as required. In addition, SpiderPool will verify the addresses in the IP Pool instance and ensure that the IP addresses do not overlap.
The CR design of IP Pool is fully based on the scarcity of IP addresses, realizes the binding of different IP Pools in applications and namespaces to realize IP resource isolation, and also meets the scenario of IP resource sharing.
1. SpiderSubnet object: A Subnet corresponds to an IPv4/IPv6 subnet, which stores all available IP addresses under the subnet.
2. SpiderIPPool object: There can be multiple SpiderIPPool instances under a Spider Subnet object, and the SpiderIPPool object stores part of the available IP address collection of the SpiderSubnet it belongs to. Pods use IP addresses assigned from the Spider IP Pool object.
Related CRD Resources
1. spidersubnets.spiderpool.spidernet.io is used to store and define IP subnet instances. It stores all available IP collections under the subnet, and the IP in the IP Pool is obtained from the IP subnet.
Check the rationality: The Webhook mechanism ensures that when the administrator creates and modifies the SpiderSubnet object, it will not overlap with the subnet of the stock Spider Subnet object, and ensures that the values of all fields in it are in compliance with the specification.
Update the managed SubnetIPPool data in real time: subnetIPPool Informer will update the data to the managed SpiderSubnet object in real time.
Processing object deletion: When the SpiderSubnet object is deleted, the Finalizer ensures that the IPs in all managed SpiderIPPools are released by the business Pod, and then the SpiderSubnet object is actually deleted, avoiding IP leaks and conflicts.
Status display: Display the usage information of the Status field.
apiVersion: spiderpool.spidernet.io/v1kind: SpiderSubnetmetadata:finalizers:- spiderpool.spidernet.ioname: default-v4-subnetspec:ipVersion: 4subnet: 172.19.0.0/16ips:- 172.19.40.2-172.19.40.254excludeIPs:xxxxgateway: xxxxroutes: xxxxvlan: 0status:totalIPCount: 253allocatedIPCount: 253controlledIPPools:default-v4-ippool:ips:- 172.19.40.2-172.19.40.254
2. spiderippools.spiderpool.spidernet.io stores part of the available IP collection under the subnet. The available IPs in each IP Pool are defined in the CR instance.
Check the rationality: When creating and modifying SpiderIPPool objects, the Webhook mechanism ensures that the values of all fields in it are in compliance with the specification, and does not allow IP overlap between SpiderIPPool objects.
Guarantee subnet ownership: ensure that the spec.subnet field of SpiderIPPool must correspond to a stock SpiderSubnet object, and ensure that the available IP addresses in spec.ips must be idle IP addresses in the SpderSubnet object to which it belongs.
Process object deletion: Finalizer intercepts the deletion of SpiderIPPool objects. Only after all the IPs in the SpiderIPPool object are released by the relevant business Pods, the SpiderIPPool instance will be deleted in order to avoid concurrent creation of other SpiderIPPool objects and introduce IP overlap. When the SpiderIPPool object is in the Deleting state, IPAM will stop allocating IP addresses from the Spiderippool object for new PODs, but allow the existing Pods to release the IPs.
Status display: update the status information in the Status field in real time.
apiVersion: spiderpool.spidernet.io/v1kind: SpiderIPPoolmetadata:name: default-v4-ippoolfinalizers:- spiderpool.spidernet.iospec:disable: falseipVersion: 4subnet: 172.19.0.0/16ips:- 172.19.40.2-172.19.40.254 excludeIPs: 172.19.40.254gateway: 172.19.40.1namespaceAffinity: xxxxxnodeAffinity: xxxxxpodAffinity: xxxxxroutes: xxxxvlan: 0status:allocatedIPCount: 4allocatedIPs:172.19.40.21:containerID: ****pod: coredns-6d4b75cb6d-627xrnamespace: kube-systeminterface: eth0node: spider-control-planeownerControllerName: coredns-6d4b75cb6downerControllerType: ReplicaSettotalIPCount: 253
3. spiderendpoints.spiderpool.spidernet.io records the current and historically assigned IP information of a Pod.
The status.current field stores the current latest IP allocation of the Pod. This field can store the IP allocation of all NICs to support the Pod multi-NIC scenario.
The status.history field stores all historical IP allocations, which can be used for IP GC recycling of Spiderippool objects.
The design of ownerReferences ensures the recycling mechanism of this object.
The finalizer is designed to ensure that the information stored by the object is not accidentally lost.
apiVersion: spiderpool.spidernet.io/v1kind: SpiderEndpointmetadata:finalizers: [“spiderpool.spidernet.io”]name: test-pod-788d579c4-6r2w7namespace: defaultownerReferences:- apiVersion: v1kind: Podname: test-pod-788d579c4-6r2w7uid: ca1263c8-c4ed-402c-b25a-ad3f822605f9status:ownerControllerName: test-pod-788d579c4ownerControllerType: ReplicaSetcurrent:containerID: ******creationTime: "2022-10-20T09:10:45Z"node: spider-workerips:- interface: eth0ipv4: 172.19.40.47/16ipv4Pool: default-v4-ippoolipv6: fc00:f853:ccd:e793:f::3c/64ipv6Pool: default-v6-ippoolvlan: 0- interface: net1ipv4: 172.19.40.167/16ipv4Pool: default-v4-ippoolipv6: fc00:f853:ccd:e793:f::c5/64ipv6Pool: default-v6-ippoolvlan: 0history: *******
4. spiderreservedips.spiderpool.spidernet.io is used to store IPs that need to be reserved. Once reserved, IPs cannot be allocated to Pods.
04
Future Outlook of SpriderPool
The core functions of SpiderPool are basically ready. It has been tested and implemented based on Underlay CNI combinations such as Macvlan /SRI-OV, and combined with Multus and Overlay CNI (Calico, Cilium) to realize IP allocation and management in multi-network card scenarios. In the next stage, SpiderPool will continue to enrich various usage scenarios, improve logs, and gradually enter a stable period of functions.
Cloud native technology enthusiasts are welcome to download and use, participate in community discussions and contributions.
SpiderPool official community:
https://github.com/spidernet-io/spiderpool