Didi has open sourced its Kafka monitoring and control platform Logi-KafkaManager. It has used kafka-manager, kafka-eagle, and kafka-mirrorkaker before. If there are tools that can integrate the advantages of these tools, it is for the production environment The management, monitoring, and resource allocation of the Kafka cluster are very good, so while studying the source code of Kafka, study the source code and usage of Logi-KafkaManager. Didi provides the experience address: http://117.51.150.133:8080/kafka , Account admin/admin .
Reference materials:
One, source code interpretation
1. Front-end debugging environment
Github cloning is slower and gitee is fast. It adopts a front-end and back-end separation architecture (springboot+reactJS+Typescript). The code contains several modules common, console, core, dao, extends, task, web, among which the web has the start of the MainApplication project. Class, others are dependent. The console module is based on the front-end interface of recat+typescript (the technology stack selection is still very advanced), and the source code of the front-end and back-end runs are viewed locally. Here, the console is run separately in VScode;
# react跟vue一样基于node,所以npm相关依赖引入和配置启动
npm config set registry https://registry.npm.taobao.org
npm config list #查看npm当前配置
npm install
# 启动react项目
npm start
Start the console front-end module:
pom.xml needs to comment out the reference to the console front-end module:
2. Post-debugging environment
Depends on Maven 3.5+ (back-end packaging), node v12+ (front-end packaging), Java 8+ (required for operating environment), MySQL 5.7 (data storage), node is not needed because it is placed in vscode, create kafka_manager library in mysql, and Run the sql initialization statement and modify the mysql configuration in springboot at the same time;
mysql --default-character-set=utf8 -uroot -p123456 -P3306 -D kafka_manager < create_mysql_table.sql
Configure MainApplication.java of the web module as the main application class to start;
2021-01-25 19:33:22.642 INFO 18000 --- [ main] c.x.kafka.manager.web.MainApplication : MainApplication started
Because it is running locally, the proxy target of the console module needs to be modified:
proxy: {
'/api/v1/': {
target: 'http://127.0.0.1:8080',
//target: 'http://10.179.37.199:8008',
// target: 'http://99.11.45.164:8888',
changeOrigin: true,
}
Above, the debugging environment based on front-end and back-end separation is run independently locally; you can see that the front-end reads the Kafka cluster configuration in the mysql library;
3. Back-end module architecture
Refer to the official application architecture diagram to analyze the code architecture;
+----------------------------+
| Tables_in_kafka_manager |
+----------------------------+
| account |
| app |
| authority |
| broker |
| broker_metrics |
| cluster |
| cluster_metrics |
| cluster_tasks |
| config |
| controller |
| gateway_config |
| heartbeat |
| kafka_acl |
| kafka_bill |
| kafka_file |
| kafka_user |
| logical_cluster |
| monitor_rule |
| operate_record |
| reassign_task |
| region |
| topic |
| topic_app_metrics |
| topic_connections |
| topic_expired |
| topic_metrics |
| topic_report |
| topic_request_time_metrics |
| topic_statistics |
| topic_throttled_metrics |
| work_order |
+----------------------------+
31 rows in set (0.01 sec)
2. Tool understanding
Three, deployment verification