Istio Fault Injection and Retry Experiment

fault injection

---

Istio traffic management has the function of fault injection. When receiving the traffic of the user request program, it injects fault phenomena, such as injecting HTTP request errors. When traffic enters Sidecar, it directly returns a 500 error request code.

Fault injection can be used to test the fault recovery capability of the entire application, inject various fault phenomena, and take countermeasures for different fault phenomena.

There are two types of fault injection:

• Delay: Injecting a delay fault can simulate a phenomenon that the system responds very slowly when the load is high.
• Abort: Injecting an abort fault can simulate a system crash and directly return an HTTP error code or TCP connection failure

When using fault injection, the configuration of enabling timeout and retry is not available, and the fault injection is configured in the VirtualService resource.

List of configurations for two types of fault injection:

• Fault Injection - Latency Configuration Checklist

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: tomcat-vs
  namespace: istio-traffic 
spec:
  http:
  - fault:							#定义故障注入
      delay:						#定义故障注入的延迟配置
        percentage:						#配置要将流量注入故障的比例，这里为100%，也可以针对50%的流量进行故障注入，剩下50%的流量做一些其他路由匹配				
          value: 100
        fixedDelay: 10s					#故障注入延迟响应时间10秒
    route:						#将满足的流量路由到tomcat的service资源上
    - destination:
        host: tomcat-svc

注意：
	1.falut和route是同级参数，当满足故障注入条件的流量会被route进行路由分发。
	2.重试和超时的配置都是在route中设置的，当我们配置了故障注入后就无法再配置超时和重试了。

• Fault Injection-Abort Configuration Checklist

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: tomcat-vs
  namespace: istio-traffic 
spec:
  http:
  - fault:							#定义故障注入
      abort:						#定义故障注入的中止配置
        percentage:						#配置要将流量注入故障的比例，这里为100%，也可以针对50%的流量进行故障注入，剩下50%的流量做一些其他路由匹配				
          value: 100

        httpStatus: 503
 					#注入的故障为返回一个503的HTTP状态码
    route:						#将满足的流量路由到tomcat的service资源上
    - destination:
        host: tomcat-svc

Retry

---

Retrying is very understandable, if the request times out then it will be retried.

In the Istio service grid, the Envoy proxy will not try to reconnect to the service after the request fails or times out, and will not retry. We need to configure Istio's retry mechanism. Retry and timeout policies can be used simultaneously.

 Retry configuration list in VirtualService resource:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - route:
    - destination:
        host: ratings
        subset: v1
    retries:						#定义重试策略
      attempts: 3
					#重试的次数为3次
      perTryTimeout: 2s					#重试的间隔为2秒

In the Istio service grid, the Envoy proxy will not try to reconnect to the service after the request fails or times out, and will not retry. We need to configure Istio's retry mechanism.

Retry and timeout policies can be used simultaneously.

Retry configuration list in VirtualService resource:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - route:
    - destination:
        host: ratings
        subset: v1
    retries:						#定义重试策略
      attempts: 3                   #重试的次数为3次
      perTryTimeout: 2s				#重试的间隔为2秒

 

 

 

Configure fault injection and retries for applications in Istio

---

case description

 

Case: Nginx reverse proxy Tomcat service, the VirtualService of Nginx service starts the retry mechanism, the VirtualService of Tomcat service configures fault injection, never responds to the request of Nginx service, and observes whether the retry mechanism will be triggered.

Roughly realize the idea:

• Deploy an Nginx service and a Tomcat service in Istio, and configure Nginx reverse proxy Tomcat service.

• Set the number of retries for the Nginx service through VirtualService, and retry when the request fails.

• The Tomcat service returns a 503 error code through the fault injection configuration in the VirtualService and any request will fail.

• At this time, when Nginx goes to reverse proxy Tomcat, Tomcat will only return a 503 error code, and observe whether there is a retry record.

Create a Namespace and enable Sidecar automatic injection.

[root@k8s-master istio-traffic]# kubectl label ns istio-traffic istio-injection=enabled
namespace/istio-traffic labeled

[root@k8s-master istio-traffic]# kubectl create ns istio-traffic
namespace/istio-traffic created

 Deploy Nginx service in Istio

1) Write resource orchestration files

[root@k8s-master istio-traffic]# vim nginx-k8s.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-proxy
  namespace: istio-traffic
  labels:
    server: nginx
    app: web
spec:
  replicas: 
1

  selector:
    matchLabels:
      server: nginx
      app: web
  template:
    metadata:
      name: nginx
      labels: 
        server: nginx
        app: web
    spec:
      containers:
      - name: nginx
        image: jiangxlrepo/know-system:v1		#knowsystem镜像中包含了Nginx
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: istio-traffic
spec:
  selector:
    server: nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

 2) Create a resource and view the status of the resource

1.创建资源
[root@k8s-master istio-traffic]# kubectl apply -f nginx-k8s.yaml
deployment.apps/nginx-proxy created
service/nginx-svc created

2.查看资源的状态
[root@k8s-master istio-traffic]# kubectl get all -n istio-traffic 
NAME                               READY   STATUS    RESTARTS   AGE
pod/nginx-proxy-7c487c794d-wt6nq   2/2     Running   0          53s

NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/nginx-svc   ClusterIP   10.105.3.226   <none>        80/TCP    53s

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-proxy   1/1     1            1           54s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-proxy-7c487c794d   1         1         1       54s

2.3. Deploy Tomcat service in Istio

1) Write resource orchestration files

[root@k8s-master istio-traffic]# vim tomcat-k8s.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat
  namespace: istio-traffic
  labels:
    server: tomcat
    app: web
spec:
  replicas: 
1

  selector:
    matchLabels:
      server: tomcat
      app: web
  template:
    metadata:
      name: tomcat
      labels: 
        server: tomcat
        app: web
    spec:
      containers:
      - name: tomcat
        image: docker.io/kubeguide/tomcat-app:v1 
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-svc
  namespace: istio-traffic
spec:
  selector:
    server: tomcat
  ports:
  - name: http
    port: 8080
    targetPort: 8080
    protocol: TCP

2) Create a resource and view the status of the resource

1.创建资源
[root@k8s-master istio-traffic]# kubectl apply -f tomcat-k8s.yaml 
deployment.apps/tomcat created
service/tomcat-svc created

2.查看资源的状态
[root@k8s-master istio-traffic]# kubectl get all -n istio-traffic 
NAME                               READY   STATUS    RESTARTS   AGE
pod/nginx-proxy-7c487c794d-wt6nq   2/2     Running   0          6m1s
pod/tomcat-86ddb8f5c9-7n2xj        2/2     Running   0          16s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/nginx-svc    ClusterIP   10.105.3.226   <none>        80/TCP     6m1s
service/tomcat-svc   ClusterIP   10.100.46.31   <none>        8080/TCP   16s

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-proxy   1/1     1            1           6m1s
deployment.apps/tomcat        1/1     1            1           16s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-proxy-7c487c794d   1         1         1       6m1s
replicaset.apps/tomcat-86ddb8f5c9        1         1         1       16s

2.4. Configure Nginx reverse proxy Tomcat

1.配置Nginx反向代理
[root@nginx-proxy-76ccd8b9fc-8hqbq /]# vim /data/nginx/conf/conf.d/istio-tomcat.conf
server {
	listen 80;
	server_name _;

  	location / {
		proxy_pass http://tomcat-svc:8080;
		proxy_http_version 1.1;
  	}
}

2.重载Nginx配置
[root@nginx-proxy-76ccd8b9fc-8hqbq /]# /data/nginx/sbin/nginx -t
nginx: the configuration file /data/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /data/nginx/conf/nginx.conf test is successful
[root@nginx-proxy-76ccd8b9fc-8hqbq /]# /data/nginx/sbin/nginx -s reload

2.5. Configure retries in the Nginx service VirtualService resource

Configure the VirtualService resource of the Nginx service and set the timeout period.

1.编写资源编排文件
[root@k8s-master istio-traffic]# vim nginx-vs-att.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: nginx-vs
  namespace: istio-traffic 
spec:
  hosts:
  - nginx-svc					#虚拟主机列表中指定了Nginx的Service资源名称，用于内部调用，接收来自nginx-svc service资源的流量
  http:
  - route:						#将流量转发到nginx-svc Service资源上
    - destination: 
        host: nginx-svc				
    retries:					#定义重试策略
      attempts: 3           	#重试次数为3次
      perTryTimeout: 2s			#重试的间隔为2秒

 
2.创建资源编排文件
[root@k8s-master istio-traffic]# kubectl apply -f nginx-vs-att.yaml
virtualservice.networking.istio.io/nginx-vs created

3.查看资源
[root@k8s-master istio-traffic]# kubectl get vs -n istio-traffic 
NAME       GATEWAYS   HOSTS           AGE
nginx-vs              ["nginx-svc"]   58s

2.6. Configuring Fault Injection in Tomcat Service VirtualService Resource

Configure the VirtualService resource of the Nginx service, set the delay time of the request response, and set the delay time a little longer. After this period of time, return the response to achieve the effect of timeout configuration.

1.编写资源编排文件
[root@k8s-master istio-traffic]# vim tomcat-vs-att.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: tomcat-vs
spec:
  hosts:
  - tomcat-svc				#虚拟主机列表中指定了Tomcat的Service资源名称，用于内部调用，接收来自tomcat-svc service资源的流量
  http:
  - fault:							#定义故障注入策略
      abort:						#定义中止类型的故障注入
        percentage:						#配置要将流量注入故障的比例，这里为100%，也可以针对50%的流量进行故障注入，剩下50%的流量做一些其他路由匹配				
          value: 100
        httpStatus: 503				#注入的故障为返回一个503的HTTP状态码
    route:						#将满足的流量路由到tomcat的service资源上
    - destination:
        host: tomcat-svc
 
2.创建资源编排文件
[root@k8s-master istio-traffic]# kubectl apply -f tomcat-vs-att.yaml
virtualservice.networking.istio.io/tomcat-vs created

3.查看资源
[root@k8s-master istio-traffic]# kubectl get vs -n istio-traffic
NAME        GATEWAYS   HOSTS            AGE
nginx-vs               ["nginx-svc"]    6m42s
tomcat-vs              ["tomcat-svc"]   4s

3. Verify the traffic retry effect of Istio configuration

The Tomcat service is configured with fault injection, and all traffic requests will return a 503 error code. Nginx forwards the request to Tomcat, but Tomcat will not respond to the Nginx request. At this time, the configured retry mechanism will be triggered, and a retry will be performed every two or two seconds. After three retries, it will exit.

[root@k8s-master istio-traffic]# kubectl run -it  busybox --image busybox:1.28 --restart=Never --rm busybox -n istio-traffic -- sh
/ # wget -q -O - http://nginx-svc
wget: server returned error: HTTP/1.1 503 Service Unavailable
/ # wget -q -O - http://nginx-svc
wget: server returned error: HTTP/1.1 503 Service Unavailable
/ # wget -q -O - http://nginx-svc
wget: server returned error: HTTP/1.1 503 Service Unavailable

We can request the Nginx service in the busybox container, and Nginx forwards the request to Tomcat. Since Tomcat cannot respond, Nginx will retry 3 times after sending a request. When we execute a wget request, 4 request records will be generated, and these 4 request records will generate 8 logs of the Envoy proxy program, which are logs of incoming and outgoing traffic. We can clearly see in the log that there will be 2 log records for each request or retry.

When we see 3 retries associated with a request in the log, it means that the retry configuration has taken effect.

[root@k8s-master ~]# kubectl logs -f nginx-proxy-76ccd8b9fc-8hqbq -c istio-proxy -n istio-traffic