12.23 Yuanchuanghui·Shanghai Station, let’s talk about LLM infrastructure

This article is shared from the Huawei Cloud Community " Cloudeye connects with Prometheus to implement all-round monitoring of Huawei Cloud ". The author: You can make a friend.

1. Background

The Cloudeye service provides us with a three-dimensional monitoring platform for elastic cloud servers, broadband and other resources. Help us fully understand the resource usage and business operating status on Huawei Cloud. After connecting Huawei Cloud Cloudeye service to prometheus, you can use the Prometheus ecosystem to better monitor and analyze data from the Cloudeye service.

2. Introduction

By using cloudeye-exporter, connect the Huawei Cloud Cloudeye service with the Prometheus container in the Huawei Cloud CCE cluster (Kubernetes cluster). Then use the Prometheus system to perform operation and maintenance monitoring of the performance data of cloud resources.

Note: This exporter is only applicable to Huawei Cloud locations.

3. Practical Demonstration

Prometheus will be deployed in the kubernetes cluster as an operator. Therefore, it is also necessary to create a cloudeye-exporter container image, and then containerize and deploy it in the kubernetes cluster.

3.1 Obtain the cloudeye-exporter installation package

1. Download the latest version of cloudeye-exporter

Check the plug-in Releases version ( https://github.com/huaweicloud/cloudeye-exporter/releases ), obtain the cloudeye-exporter download address, download, decompress and install.

mkdir cloudeye-exporter & cd cloudeye-exporter

wget https://github.com/huaweicloud/cloudeye-exporter/releases/download/v2.0.6/cloudeye-exporter.v2.0.6.tar.gz

2. Decompress

tar -xzvf cloudeye-exporter.v2.0.6.tar.gz

3.2 Making container images

1. You need to write a Dockerfile file. Centos is used as the base image here.

FROM centos  
COPY cloudeye-exporter logs.yml metric.yml /root/
WORKDIR /root
CMD ["/root/cloudeye-exporter"]

2. Build container images

docker build -t swr.cn-north-4.myhuaweicloud.com/k8s-solution/cloudeye-exporter:v2.0.6 --platform=linux/amd64 .

3. Push to the mirror warehouse

It is recommended to use Huawei Cloud SWR mirror warehouse

3.3 Configure cloudeye-exporter access authentication

1. Edit the clouds.yml file to configure public cloud information

When cloudeye-exporter is started, this configuration file needs to be used for access authentication of cloud services.

global: 
  port: ":8087" # Listening port: 8087 means opening the listening port 8087 on all network interfaces. You can also specify the IP to limit internal access. For example: 192.168.1.100:8087 scrape_batch_size 
  : 300 
auth: 
  auth_url: "https:// iam.cn-north-4.myhuaweicloud.com/v3" 
  project_name: "cn-north-4" # Huawei Cloud project name, which can be viewed in "Huawei Cloud->Unified Identity Authentication Service->Project" 
  access_key: "" # You can refer to Chapter 3.1 for the IAM user access key and use the command line to enter the encrypted ak sk to avoid configuring the AK SK in plain text in the configuration file 
  secret_key: "" 
  region: "cn-north-4" # Region ID

2. Create kubernetes secrets based on this configuration file

kubectl create secret generic clouds-exporter --from-file=clouds.yml -n monitoring
Containerized deployment needs to mount this configuration to cloudeye-exporter in the form of secrets

3.4 Deploy cloudeye-exporter in CCE cluster (k8s cluster)

1. First deploy the load and service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudeye-exporter
  namespace: monitoring
  labels:
    app: cloudeye-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cloudeye-exporter
  template:
    metadata:
      labels:
        app: cloudeye-exporter
    spec:
      volumes:
        - name: cloudeye
          secret:
            secretName: clouds-exporter
      imagePullSecrets:
        - name: default-secret
      containers:
      - name: cloudeye-exporter
        image: swr.cn-north-4.myhuaweicloud.com/k8s-solution/cloudeye-exporter:v2.0.6
        ports:
          - containerPort: 8087
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 500Mi
        volumeMounts:
          - name: cloudeye
            mountPath: "/root/clouds.yml"
            subPath: clouds.yml

---
apiVersion: v1
kind: Service
metadata:
  name: cloudeye-exporter
  namespace: monitoring
  labels: 
    app: cloudeye-exporter
spec:
  selector:
    app: cloudeye-exporter
  ports:
  - name: ces
    targetPort: 8087  #业务监听端口
    port: 8087
    protocol: TCP
  type: ClusterIP

2. Confirm whether cloudeye-exporter exposes indicators.

Deployed successfully, check Pod information. Access the pod indicator endpoint to confirm whether the indicator source can expose indicators.

curl http://podIP:8087/metrics?services='SYS.VPC'

The cloudeye-exporter Pod instance serves as the source of Prometheus crawling indicators and can expose monitoring data.

Check the Service to confirm whether data can be obtained by accessing the Service.

2. Pay attention

Note: cloudeye-exporter needs to access the public network endpoint: iam.huaweicloud.com . If the kuberntes platform you are using is a CCE turbo cluster, you need to configure public network snat for the container network segment. Otherwise, the indicator data capture will fail.

3.5 Prometheus connects to cloudeye-exporter to obtain indicators

Modify the prometheus.yml file configuration in prometheus. Because Prometheus is now deployed in the prometheus-operator form in k8s clusters. So we need to write ServiceMonitor to obtain indicators.

1. Take monitoring Huawei Cloud VPC and ELB services as an example

The capture rules cloudeye-exporter-servicemonitor.yaml are as follows:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: 
  name: cloudeye-exporter
  namespace: monitoring 
  labels:
    app: cloudeye-exporter
spec:
  endpoints:
    - interval: 1m
      scrapeTimeout: 1m
      port: ces
      scheme: http
      params:
        services: ['SYS.VPC,SYS.ELB']
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app: cloudeye-exporter

2. Create the above crawling rules in the cluster. Wait for a moment to observe that Prometheus has successfully crawled

3. At this point, cloudeye-exporter has successfully connected to Prometheus.

3.6 Connect to Grafana market

Huawei Cloud Community provides several ready-made dashboards for cloud services. We can use it.

https://github.com/huaweicloud/cloudeye-exporter/tree/v2.0.6/grafana_dashboard/templates

For example: use the dashboard of VPC cloud service.

1. Upload the json file of the dashboard to grafana

After selecting the data source, click Import.

2. Detect abnormalities

You will find that the dashboard does not generate data

3. Solution

According to Huawei Cloud Community Statement: If you need to use the following template directly, you need to modify the prometheus configuration and add the task of obtaining enterprise project information. So we also need to modify the servicemonitor and add a configuration based on the original cloudeye-exporter-servicemonitor.

apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
metadata: 
  name: cloudeye-exporter 
  namespace: monitoring 
  labels: 
    app: cloudeye-exporter 
spec: 
  endpoints: 
    - interval: 1m 
      scrapeTimeout: 1m 
      port: ces 
      scheme: http 
      params: 
        services: ['SYS.VPC,SYS.ELB'] 
    # Configure this task to obtain enterprise project information for the dashboard of Huawei Cloud Service 
    - interval: 1m 
      scrapeTimeout: 1m 
      port: ces 
      path: "/eps-info" 
      scheme: http 
  namespaceSelector: 
    matchNames: 
    - monitoring 
  selector: 
    matchLabels: 
      app: cloudeye-exporter