This article is shared from the Huawei Cloud Community " Performing HPA on the specified backend based on Nginx Ingress indicators ". The author: You can make friends.
background
In production scenarios, Nginx Ingress traffic will eventually be forwarded to different applications through domain names and paths. Sometimes CPU and memory are not the performance bottlenecks of nginx. In this case, nginx_ingress_controller_requestsHPA can be configured for the corresponding application based on indicators to achieve Elastically specify back-end workloads based on the request volume of different domain names and paths.
Introduction
Environmental preparation
- nginx-ingress has been deployed
- The cloud native monitoring plug-in kube-prometheus-stack has been installed (server mode). The plug-in monitors nginx-ingress by default. For open source environments, please configure monitoring by yourself.
- Configured kubectl command or use cloudshell
Note: Since the namespace cannot be specified in the HPA rules scaleTargetRefand describedObjectthe two fields, the indicator source, HPA and elastic target need to be in the same namespace, while nginx-ingress and business workloads are generally in different namespaces; this solution uses external Type HPA, the namespace of the indicator source can be ignored
Steps
Create the elastic target workload, service and ingress required for the demo
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-hpa
labels:
app: test-app
spec:
replicas: 1
selector:
matchLabels:
app: test-app
template:
metadata:
labels:
app: test-app
spec:
containers:
- image: skto/sample-app:v2
name: metrics-provider
ports:
- name: http
containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: test-app
namespace: default
labels:
app: test-app
spec:
ports:
- port: 8080
name: http
protocol: TCP
targetPort: 8080
selector:
app: test-app
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-app
labels:
app: sample-app
spec:
replicas: 1
selector:
matchLabels:
app: sample-app
template:
metadata:
labels:
app: sample-app
spec:
containers:
- image: skto/sample-app:v2
name: metrics-provider
ports:
- name: http
containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: sample-app
namespace: default
labels:
app: sample-app
spec:
ports:
- port: 80
name: http
protocol: TCP
targetPort: 8080
selector:
app: sample-app
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-ingress
namespace: default
spec:
ingressClassName: nginx
rules:
- host: test.example.com
http:
paths:
- backend:
service:
name: sample-app
port:
number: 80
path: /
pathType: ImplementationSpecific
- backend:
service:
name: test-app
port:
number: 8080
path: /home
pathType: ImplementationSpecific
Query test.example.com/and test.example.com/homenginx_ingress_controller_requests indicators respectively, the indicators are normal
Create an apiservices resource of external type; it is normal for the status of apiservices to be false after creation. After adding externalRules, the status becomes true.
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.external.metrics.k8s.io
spec:
group: external.metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service: #Specify the service corresponding to prometheus-adapter. The adapter name in the Huawei CCE plug-in is custom-metrics-apiserver.
name: custom-metrics-apiserver
namespace: monitoring
port: 443
version: v1beta1
versionPriority: 100
Add the externalRules rules to the adapter's configmap. After modification, you need to restart the prometheus-adapter service.
kubectl -n monitoring edit configmap user-adapter-config
externalRules:
- metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)
name:
as: ${1}_per_second
matches: ^(.*)
resources:
namespaced: false #Ignore the namespace of the indicator source, this configuration does not apply rules rules
seriesQuery: nginx_ingress_controller_requests
seriesQuery : original indicator; you can write the indicator name directly, or you can use {labelKey=labelValue} to filter out the original indicator.
metricsQuery : Use PromQL to filter and aggregate indicators; .Seriesrepresents original indicators, .LabelMatchersrepresents label filtering of indicators, and specific filtering rules can be configured in hpa
name : Rename the indicator
resources : Called through API when hpa queries indicators, the calling path is:
The function of resources is to replace ${namespace} in the path with the value of the namespace tag in the indicator, and our solution needs to ignore the indicator source namespace.
After restarting the custom-metrics-apiserver service, you need to wait for about 1 minute and execute the command to check whether the indicators are normal.
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/*/nginx_ingress_controller_requests_per_second
Create HPA rules
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: sample-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: sample-app
min Replies: 1
maxReplicas: 10
metrics:
- type: External
external:
metric:
name: nginx_ingress_controller_requests_per_second
selector:
matchLabels: #You can filter indicators through this field. The label filtering conditions here will be added to <<.LabelMatchers>> of externalRules.
exported_service: sample-app #Filter requests whose backend service is sample-app
host: test.example.com #Filter requests to access the domain name test.example.com
target:
type: AverageValue #External indicator type only supports target values of Value and AverageValue types.
averageValue: 30
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: test-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-app
min Replies: 1
maxReplicas: 10
metrics:
- type: External
external:
metric:
name: nginx_ingress_controller_requests_per_second
selector:
matchLabels:
exported_service: test-app
host: test.example.com
target:
type: AverageValue
averageValue: 30
Flexible demo
Use the command to stress test the access domain name and path corresponding to the sample-app, and trigger the elasticity normally; please configure the mapping between the domain name and the ELB address yourself.
ab -c 50 -n 5000 http://test.example.com/
Use the same method to stress-test test-app and trigger elasticity normally.
ab -c 50 -n 5000 http://test.example.com/home
Click to follow and learn about Huawei Cloud’s new technologies as soon as possible~
Linus took it upon himself to prevent kernel developers from replacing tabs with spaces. His father is one of the few leaders who can write code, his second son is the director of the open source technology department, and his youngest son is an open source core contributor. Robin Li: Natural language will become a new universal programming language. The open source model will fall further and further behind Huawei: It will take 1 year to fully migrate 5,000 commonly used mobile applications to Hongmeng. Java is the language most prone to third-party vulnerabilities. Rich text editor Quill 2.0 has been released with features, reliability and developers. The experience has been greatly improved. Ma Huateng and Zhou Hongyi shook hands to "eliminate grudges." Meta Llama 3 is officially released. Although the open source of Laoxiangji is not the code, the reasons behind it are very heart-warming. Google announced a large-scale restructuring