In the previous content, we took a deep dive into how EasyMR leverages Kubernetes for deployment. As you have learned, in the overall architecture of EasyMR, we use Prometheus to collect, query and store node and service monitoring data. At the same time, Grafana , as a powerful visualization tool, displays the monitoring data in Prometheus in a variety of ways.
In this article, we will discuss in detail how to dynamically collect Kubernetes application monitoring data in EasyMR.
Pain points of traditional collection solutions
In host mode, the configuration monitored by EasyMR using Prometheus mainly depends on static_configs and file_sd_configs. Because under this deployment scheme, the stability of nodes and applications is high, and the changes and uncertainties involved are small. Unless there is an extreme situation such as node downtime, we need to manually modify the corresponding collection job configuration.
However, in the context of the cloud native era, monitoring, as a key part of observability practice , has undergone some major changes compared to system and application monitoring under traditional architecture:
· Microservices and application containerization lead to an exponential increase in monitoring objects and indicators
· The life cycle of monitoring objects is shorter, resulting in an exponential increase in the amount and complexity of monitoring data.
In layman's terms, our Kubernetes cluster will have many Node/Service/Pod and other resources. These resources will dynamically change as the demand scale changes. The IP and name of the same application Pod will also change as the application restarts. Changes due to rolling updates.
So when Kubernetes resources are created or updated, it will be a very huge workload to modify the Job tasks in Prometheus one by one, because you cannot judge the restart time of the Pod (Kubernetes has its own scheduler, maybe the CPU/CPU of the host where the Pod is currently located The memory/disk pressure is too high, the Pod may reach the set resource limit, etc., which will cause the Pod to be rescheduled). In this context, we need Prometheus to have the function of automatic service discovery .
Prometheus service automatic discovery
For the above scenario where static_configs and file_sd_configs cannot be configured using static collection, Prometheus itself provides a solution: introducing a service registration center . This registration center holds the access information of all current monitoring targets, and Prometheus only needs to ask it which monitoring targets there are. Prometheus queries the target list that needs to be monitored , and then polls these targets to obtain monitoring data.
Prometheus supports multiple service discovery mechanisms : files, DNS, Consul, Kubernetes, OpenStack, EC2, etc. This article takes the Kubernetes service discovery mechanism as an example to discuss in detail.
Under Kubernetes, Prometheus mainly supports five service discovery modes through integration with the Kubernetes API: Node, Service, Pod, Endpoints, and Ingress.
Different service discovery modes are suitable for different scenarios. For example: Node is suitable for monitoring resources related to the host, such as the status of Kubernetes components running in the node, the status of the containers running on the node, etc.; Service and Ingress are suitable for black box monitoring . Scenarios, such as monitoring service availability and service quality; Endpoints and Pods can both be used to obtain monitoring data of Pod instances , such as monitoring applications deployed by users or administrators that support Prometheus.
EasyMR’s monitoring practice for K8S applications
Prometheus uses pull mode to obtain metrics, so the /metrics interface needs to be exposed for target applications that need to be monitored. Next, we start with the steps of deploying Prometheus and collect MySQL monitoring information to describe in detail how EasyMR uses Prometheus' service discovery mechanism.
Create Prometheus configuration file
The core of Prometheus' automatic discovery lies in the related configuration of relabel_configs. First, configure these metadata labels starting with _ meta through source_labels, declare the resources to be matched, then find the relevant resource objects through regex matching rules , and finally collect them. Perform secondary processing on the indicators, such as retaining, transferring, replacing and other operations.
Create ServiceAccount/Role/RoleBinding
Since Prometheus needs to access Kubernetes resources, and Kubernetes has a detailed RBAC permission control mechanism , you need to create a corresponding account before deploying Prometheus and grant the account permissions to the corresponding interface. For security reasons, EasyMR only needs to obtain the permissions of the corresponding namespace, so we do not need the global ClusterRole permissions.
Create Prometheus Deployment
Since Prometheus needs to store data, the corresponding PV needs to be created in advance. The official recommendation is to use localpv, which will not be described here. In the Deployment configuration file, we only need to specify the PVC name. The key configuration is shown here.
部署 MySQL Statefulset、MySQL Exporter、MySQL Service
Since the same Pod shares network and storage, we deploy MySQL Exporter as a separate Container and MySQL Container in the same Pod in the deployment architecture. We only need to expose the monitoring port of MySQL Exporter to the Prometheus registration center. That’s it, some important configurations are as follows:
● MySQL Statefulset
● MySQL Service
Add two configurations under the annotations of the Service configuration:
· prometheus.io/port: 9104
· prometheus.io/scrpae: true
View Prometheus Targets configuration
You can see that Prometheus has dynamically discovered the monitoring data exposed by the deployed MySQL service, and the status is UP, without manual intervention.
View the EasyMR Grafana dashboard
After the above operations, we can easily see rich MySQL monitoring information on the EasyMR page, and other services can also be completed through similar steps.
Conclusion
In the cloud native era, the combination of Promtheus+Grafana has become an indispensable part of the observability tools , but how to maximize their role still requires everyone to explore in depth. In the future, EasyMR will also make its own explorations in other fields of observability (logging, tracing).
"Dutstack Product White Paper" download address: https://www.dtstack.com/resources/1004?src=szsm
"Data Governance Industry Practice White Paper" download address: https://www.dtstack.com/resources/1001?src=szsm
For those who want to know or consult more about big data products, industry solutions, and customer cases, visit the Kangaroo Cloud official website: https://www.dtstack.com/?src=szkyzg
Broadcom announced the termination of the existing VMware partner program . Site B crashed twice, Tencent's "3.29" level one incident... Taking stock of the top ten downtime incidents in 2023, Vue 3.4 "Slam Dunk" released, Yakult confirmed 95G data Leaked MySQL 5.7, Moqu, Li Tiaotiao... Taking stock of the (open source) projects and websites that will be "stopped" in 2023 "2023 China Open Source Developer Report" is officially released Looking back at the IDE 30 years ago: only TUI, bright background color …… Julia 1.10 officially released Rust 1.75.0 released NVIDIA launched GeForce RTX 4090 D specially for sale in China