Reprinted: https://juejin.cn/post/6844903927318577159
Background introduction
In some cases, we are using Kubernetes as the cloud platform for business applications. We want to implement blue-green deployment of applications to iterate application versions. Using lstio is too heavy and complicated, and it is positioned in flow control and grid governance; Ingress -Nginx introduced the Canary function in version 0.21, which can configure multiple versions of applications for the gateway entrance, and use annotations to control the traffic distribution of multiple back-end services
Ingress-Nginx-Annotation Canary function introduction
If you want to enable the Canary function, you must first set it
nginx.ingress.kubernetes.io/canary: "true"
, and then you can enable the following annotations to configure Canary
nginx.ingress.kubernetes.io/canary-weight
The percentage of requests to the service specified in Canary Ingress. The value is an integer from 0-100. According to the set value, the percentage of traffic will be allocated to the backend service specified in Canary Ingress.nginx.ingress.kubernetes.io/canary-by-header
Traffic segmentation based on the request header is suitable for grayscale publishing or A/B testing. When the set hearer value is always, the request traffic will always be allocated to the Canary entrance. When the hearer value is set to never, the request traffic Will not be assigned to the Canary entry, for other hearder values, will be ignored, and request traffic will be assigned to other rules by prioritynginx.ingress.kubernetes.io/canary-by-header-value
This configuration shouldnginx.ingress.kubernetes.io/canary-by-header
be used together. When the hearer key and valuenginx.ingress.kubernetes.io/canary-by-header
nginx.ingress.kubernetes.io/canary-by-header-value
in the request match, the request traffic will be allocated to the Canary Ingress entry. For any other hearer values, it will be ignored, and the request traffic will be assigned to other rules by prioritynginx.ingress.kubernetes.io/canary-by-cookie
This configuration is based on cookie traffic segmentation. It is also suitable for grayscale publishing or A/B testing. When the cookie value is set to always, the request traffic will be routed to the Canary Ingress entrance. When the cookie value is set to never, the request traffic will be routed to the Canary Ingress entry. Will not be routed to the Canary entry, for other values, will be ignored, and request traffic will be assigned to other rules by priority
The canary rules are sorted by priority as follows: canary-by-header -> canary-by-cookie -> canary-weight
1. Small-scale version testing based on weights
- v1 version layout file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
labels:
app: echoserverv1
name: echoserverv1
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv1
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv1
namespace: echoserver
spec:
selector:
name: echoserverv1
type: ClusterIP
ports:
- name: echoserverv1
port: 8080
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echoserverv1
namespace: echoserver
labels:
name: echoserverv1
spec:
selector:
matchLabels:
name: echoserverv1
template:
metadata:
labels:
name: echoserverv1
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv1
ports:
- containerPort: 8080
name: echoserverv1
- View the resources created by the v1 version
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h
NAME HOSTS ADDRESS PORTS AGE
ingress.extensions/echoserverv1 echo.chulinx.com 80 24h
- Visit the v1 service, you can see that 10 requests are all access to a pod, that is, the v1 version of the service
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
- Create a v2 version of the service
We turn on the canary function and set the weight of the v2 version to 50%. This percentage does not accurately distribute requests to the two versions of the service, but fluctuates around 50%.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- View the created resource again
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
pod/echoserverv2-856bb5758-f9tqn 1/1 Running 0 4s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h
service/echoserverv2 ClusterIP 10.111.103.170 <none> 8080/TCP 4s
NAME HOSTS ADDRESS PORTS AGE
ingress.extensions/echoserverv1 echo.chulinx.com 80 24h
ingress.extensions/echoserverv2 echo.chulinx.com 80 4s
- Access test
It can be seen that 4 requests fell to the v2 version, and 6 fell to the v1 version. In theory, the more requests are said, the closer the number of requests falling to the v2 version is to the set weight of 50%.
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
2. A/B test based on header
- Change the orchestration file of the v2 version
Add header
nginx.ingress.kubernetes.io/canary-by-header: "v2"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- Update access test
Test the header for the
v2:always
v2:never
v2:true
three hearder value, as can be seen when hearderv2:always
time, all traffic will flow into v2, whenv2:never
, the traffic will flow into all v1, asv2:true
when, that is, nonalways/never
, heavy traffic will flow into the right configuration in accordance with the corresponding version of service
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
- Custom header-value
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- Update test
You can see only the header is
v2:never
at the request traffic will flow into the v2 release, other values flow will be set according to their weights inflow unreasonable version of the service
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
$ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
3. Cookie-based flow control
In fact, the principle of cookie is roughly the same as that of header. It is also the ingress automatic cookie value. If the cookie matches the client's visit, the traffic will flow to the matching back-end service.
- Update v2 version of the orchestration file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "50"
nginx.ingress.kubernetes.io/canary-by-header: "v2"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_shanghai"
labels:
app: echoserverv2
name: echoserverv2
namespace: echoserver
spec:
rules:
- host: echo.chulinx.com
http:
paths:
- backend:
serviceName: echoserverv2
servicePort: 8080
path: /
---
kind: Service
apiVersion: v1
metadata:
name: echoserverv2
namespace: echoserver
spec:
selector:
name: echoserverv2
type: ClusterIP
ports:
- name: echoserverv2
port: 8080
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoserverv2
namespace: echoserver
labels:
name: echoserverv2
spec:
template:
metadata:
labels:
name: echoserverv2
spec:
containers:
- image: mirrorgooglecontainers/echoserver:1.10
name: echoserverv2
ports:
- containerPort: 8080
name: echoserverv2
- Access test
It can be seen that the access effect is the same as that of the header, but the cookie cannot customize the value
$ [K8sSj] kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:01:52]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai:always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
# zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:02:25]
$ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai=always" echo.chulinx.com|grep Hostname;done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
to sum up
The grayscale release can ensure the stability of the overall system. The new version can be tested, found and adjusted at the initial grayscale to ensure its impact. The above content introduces Ingress-Nginx's actual combat Canary Annotation in detail through examples. You can easily achieve blue-green release and canary release with the help of Ingress-Nginx
other
About blue-green releases, canary releases, and A/B testing
- Blue-green release
In the blue-green deployment, there are two sets of systems: one is the service system, marked as "green"; the other is the system ready to be released, marked as "blue". Both systems are fully functional, and the systems that are running are only different in system versions and external services. Initially, there was no system, and there was no blue-green distinction. Then, the first system was developed and launched directly. There is only one system in this process, and there is no blue-green distinction. Later, a new version was developed, and the old version online was replaced with the new version. In addition to the online system, a brand new system using the new version code was built. At this time, a total of two systems are running. The old system that is providing services to the outside world is the green system, and the newly deployed system is the blue system. The blue system does not provide external services, what is it used for? It is used for pre-release testing. Any problems found during the testing process can be modified directly on the blue system without interfering with the system that the user is using. (Note that only when the two systems are not coupled can you be 100% guaranteed not to interfere.) After repeated testing, modification, and verification of the blue system, after confirming that it meets the online standard, the user is directly switched to the blue system: a period of time after switching Inside, the blue and green systems are still coexisting, but the user is already accessing the blue system. During this time, observe the working status of the blue system (new system), if there is a problem, switch back to the green system directly. When it is believed that the blue system that provides services to the outside world is working properly, and the green system that does not provide services to the outside world is no longer needed, the blue system officially becomes the service system to the outside world and becomes the new green system. The original green system can be destroyed to free up resources for the deployment of the next blue system. Blue-green deployment is only one of the online strategies, it is not a panacea that can deal with all situations. The premise of blue-green deployment can be implemented simply and quickly. The assumption is that the target system is very cohesive. If the target system is quite complex, then how to switch, whether the data of the two systems are required and how to synchronize, etc., all need to be carefully considered.
- Canary release
Canary release (Canary) is also a release strategy, which is the same type of strategy as the gray release that is often said in China. Blue-green deployment is to prepare two systems and switch between the two systems. The canary strategy is to only have one system and gradually replace this system. For example, the target system is a group of stateless Web servers, but the number is very large. More, suppose there are 10,000 units. At this time, blue-green deployment cannot be used, because you cannot apply for 10,000 servers specifically to deploy the blue system (in the definition of blue-green deployment, the blue system must be able to accept all access). One way that can be thought of is: Only prepare a few servers, deploy a new version of the system on them, and test and verify them. After the test passed, I was afraid of accidents, and I didn't dare to update all the servers immediately. First update 10 of the 10,000 online servers to the latest system, and then observe and verify. After confirming that there is no abnormality, update all the remaining servers. This method is canary release. In actual operation, you can also do more control. For example, set a lower weight for the 10 servers that were originally updated, control the number of requests sent to these 10 servers, and then gradually increase the weight and the number of requests. This control is called "traffic splitting", and it can be used for canary releases, and it can also be used for subsequent A/B testing. Blue-green deployment and canary release are two release strategies, and they are not omnipotent. Sometimes both can be used, sometimes only one of them can be used.
- A/B testing
The first thing to be clear is that A/B testing and blue-green deployment and canaries are completely different things. Blue-green deployment and canary is the release strategy, the goal is to ensure the stability of the newly launched system, and the focus is on the bugs and hidden dangers of the new system. A/B testing is an effect test. There are multiple versions of services for external services at the same time. These services are all services that have been tested enough to meet the online standards. There are differences but no distinction between the new and the old (they may use blue and green when they are online. Deployment method). A/B testing focuses on the actual effects of different versions of services, such as conversion rates and order status. During A/B testing, multiple versions of services are running online at the same time. These services usually have some experience differences, such as different page styles, colors, and operating procedures. Relevant personnel analyze the actual effect of each version of the service and select the version with the best effect. In the A/B test, you need to be able to control the distribution of traffic, for example, 10% of the traffic is allocated to the A version, 10% of the traffic to the B version, and 80% of the traffic to the C version.