[Jingdong Technology] Reprint must declare
This article is reproduced from IPD-Chat. IPD-Chat is the official public account of the basic platform department of Jingdong Mall. Scan the QR code to pay attention.
This article will briefly introduce the evolution of JD's message middleware. And as a message middleware in each generation of products, how do we solve some common problems faced by MQ, such as: how to deal with IO, how to store messages, how to route messages and so on.
Before we start, we need to clarify some basic concepts:
Broker: An instance of the message middleware server;
Producer: the sender of the message;
Consumer: the receiver of the message;
Topic: The label of the data classification in the middle of the message, a topic also represents a type of message, and the producer and the consumer are related through the topic.
The first generation of JMQ
In early 2012, JD.com's only product similar to message middleware was a data-based message distribution system. So to be precise, JD.com did not have a unified message middleware platform at that time. With the upgrade of the company's organizational structure, the construction of a unified message middleware platform was put on the agenda.
The goal is to build a messaging platform to provide high-availability and scalable messaging services to the outside world. and effective management of information.
First generation selection
Message core: based on ActiveMQ5.6 for extension.
Configuration Center: MySQL + ZooKeeper.
Management monitoring platform: independent research and development. Provide richer message monitoring functions than ActiveMQ's own management platform, and provide message management services
At that time, ActiveMQ was chosen because it was developed by JAVA and conformed to the company's technical route (.NET to JAVA). As for why you choose zookeeper, you can refer to "Registration Center Selection" in another article of our factory, "Service Framework Technology Selection Practice".
How AMQ solves some common problems faced by MQ:
Because the core is ActiveMQ, some of the problems come back to how ActiveMQ solves these general problems, but there are many of our extensions to ActiveMQ. For the details of ActiveMQ, we can refer to its official documents. Below we mainly look at our Extensions and improvements based on it:
1. How to store?
ActiveMQ uses the KahaDB storage engine for storage and BTree index. In order to ensure reliability, index files and log files must be refreshed synchronously. In addition, ActiveMQ implements Topics by means of virtual topics (VirtualTopic), that is, if a topic has multiple subscribers, ActiveMQ creates a queue for each subscriber, then the producer sends a message, how many subscribers, the broker will How many copies of the message will be copied and put into the queues of different subscribers, and the broker will get the message from it and push it to the consumer.
2. How to support clusters?
The native ActiveMQ client at that time was like this in sending and receiving messages: for a topic sender, it could only send messages to a specific single broker, and the same was true for consumers, which could only consume messages from a specified single broker. From the client's point of view, it does not support service clustering. Once the group of brokers assigned to this topic hangs up, the entire service will be unavailable for this topic.
The first thing we do is to let the client support the cluster. We use ZK as the configuration center to expand the client, so that the client supports the cluster and also realizes its support for the dynamic expansion of the server. The following is the overall architecture of the client and server after clustering.
Client server overall architecture
3. Push or pull? How are messages routed?
AMQ adopts the push mode, that is, after the message is sent from the producer to the broker, the broker pushes it to the consumer.
For native clients that can only send and receive messages on a single Broker, there is no problem of message routing. Let's see how messages are routed after clustering. The routing here involves two aspects, one is the producer How to choose which broker to send a message to, and how the broker decides which consumer a message should be pushed to.
Sending route: 1. Random sending, this mode is completely random, that is, the producer randomly selects a broker among the multiple brokers specified by the topic to send messages to it. 2. Weighted random transmission. In this way, we can set different weights for multiple brokers assigned to a topic. When sending, the producer selects according to the weight. The higher the weight, the greater the probability of being selected. The advantage of this method is that when a broker has an exception, we can reset its weight to 0, so that the producer will not send messages up, and after it recovers, we will gradually restore the weight, which can Avoid a large number of errors reported by the client when a single broker fails.
Consumer routing: The broker randomly selects a currently online consumer and pushes messages to it.
Client Routing Model
4. How to deal with the message of consumption failure?
The native ActiveMQ consumer will send an ACK to the broker after successfully processing a message to confirm that the message was successfully consumed. When processing a message fails, it will return a message processing failure response to the broker. At this time, when the broker receives this processing failure response, it will put the failed message in the "Dead Letter Queue" (DLQ). For each common queue, the Broker side corresponds to such a dead letter queue, and the messages of this queue cannot be acquired by consumers. In this regard, we have extended the client, intercepting the error message when there is an error in consumption, and then calling the "retry service" through the SAF protocol (our factory's SOA framework) to store the exception message into the library. After the storage is successful The consumer then sends an ACK of successful consumption to the broker, and the broker will delete the message, so we actually transfer the message from the broker side to the library. Then the consumer calls the "retry service" through SAF according to a certain strategy to obtain the previously stored messages for processing.
Client retry architecture diagram
5. How to generate message traces?
The native ActiveMQ does not support message traces. We have extended the server to intercept writing messages and consuming messages. After the messages are successfully queued and consumed, the logs are recorded respectively, and then the logs are collected and archived by the agent. The information is written into the archive library, and then the content of the message is written into JFS (JD File System, our factory's distributed file system). After the archiving is successful, the processing track of the message can be tracked on the management control platform, and the message body can be downloaded if necessary.
6. Other extensions and optimizations
Optimized the broker's message writing logic to improve performance to single-instance TPS 4000+ (native single-instance TPS3000+);
A new master-slave replication is implemented (the original master-slave replication is a complete synchronous replication, and the master data needs to be manually copied to the slave when the slave is behind, which is extremely inconvenient for operation and maintenance)
Implement a new master-slave election to make it easier to maintain and better support the cluster (the native master-slave election is prone to dual Master);
The monitoring and alarm module and other operation and maintenance tools have been added, which makes management simpler and operation and maintenance more convenient.
After a series of expansions and optimizations, AMQ basically achieved the goals of high availability, scalability, and unified governance messages, and initially formed a complete set of enterprise-level messaging middleware solutions. The following is a rough overall architecture diagram of the AMQ platform:
AMQ overall architecture
Second generation JMQ
By the beginning of 2014, AMQ has been optimized several times and has matured. However, with the development of the company's business, the number of access topics is increasing, and the amount of messages is also increasing exponentially. In addition, as the company builds new computer rooms in various places, the deployment structure of the application becomes more complicated, which leads to the requirement of cross-computer room deployment. Coupled with some problems with AMQ itself, it can be summarized as follows:
Broker is heavier and uses B-Tree index, and its performance drops sharply with the increase of message backlog;
The VirtualTopic mode adopted by the Broker side stores a message for each subscriber separately for a Topic with multiple subscribers. Most of JD's production environment uses VirtualTopic and each Topic has many subscribers. For example, the "order pipeline" message: it has nearly 100 subscribers, that is, the same data needs to be written nearly 100 copies. , not only that, but these 100 messages have to be sent to the slave through the network. After these processes, writing to TPS can only reach a few hundred. Therefore, regardless of local write performance, network utilization, or storage space utilization, this solution needs to be adjusted urgently;
Broker's logic is complex, and its model determines that individual requirements such as message playback, sequential messages, and broadcast messages cannot be extended on its basis, and in the actual use process, we are eager to support such features;
Heavy client, because functions such as clustering and abnormal message retry are supported by extending ActiveMQ's native client and introducing services such as SAF and ZooKeeper, which increases the complexity of the client to a certain extent, and the corresponding stability on the client , maintainability and other aspects have been discounted;
The registry is directly exposed to the client, so one of the most obvious drawbacks is that as the number of client instances increases, the number of connections to the registry increases, which makes it difficult to implement protection measures for the registry;
Monitoring data is not perfect.
For the above reasons, we started the self-development process of the second-generation message middleware JMQ in early 2014. Mainly do the following work:
JMQ server: implements a lightweight storage model, supports message playback, supports retry, supports message tracking, supports sequential messages, supports broadcast messages, etc., and is compatible with the OpenWire protocol used by AMQ clients;
JMQ client: realizes that the lightweight client only communicates with the Broker, supports dynamic receiving parameters, built-in performance collection, and supports cross-machine rooms;
Management control platform: more powerful management and monitoring functions;
HTTP proxy: based on Netty, supports cross-language.
Back to how JMQ solves some common problems faced by MQ:
1. How to solve the IO problem?
JMQ did not use AMQ to solve the IO problem by developing and reinventing the wheel itself, but used Netty 4.0, which is open source, supports epoll, and has a relatively simple programming model. This reduces the development work of the server to a certain extent, and also reduces the complexity of the server.
At the application layer, we have customized the JMQ protocol, and the serialization and deserialization are also completely developed by ourselves. Although this serialization and deserialization method reduces our development efficiency to a certain extent, we do not need to consider the performance loss caused by adopting a third-party serialization and deserialization scheme, and the performance improvement is obvious. of.
2. How to store messages?
JMQ storage is divided into log files (journal), message queue files (queue), and consumption location files (offset), which are all stored on the local disk of the machine where the Broker is located.
The log file mainly stores the content of the message, including the location of the queue file where the message is located. It has the following characteristics:
Different topic messages on the same broker are stored in the same log file, and the log file is divided by a fixed size;
The file name is the starting global offset;
message sequence appending;
Log files are flushed synchronously.
Since JMQ is mainly used in orders, payment and other links with extremely high reliability requirements, the broker must ensure that every message it receives falls on the physical disk. Such a log file design is mainly to improve multiple topics and large concurrent disks. write performance. Not only limited to the design of the model, in order to improve the writing performance, we also implemented Group Commit logically. The following figure is the basic schematic diagram of Group Commit in JMQ:
The message queue mainly stores the global offset of the log file where the message is located. This file has the following characteristics:
The queue information of different topics on the same broker is stored in different queue files, and the queue files are divided according to a fixed size;
The file name is the global offset;
index order append;
Queue files are flushed asynchronously.
Since the writing to the log is single-threaded, we can obtain the position of the message in the queue file in advance before writing, and write this position to the log file. Therefore, as long as the log file is successfully written, even if the queue file fails to be written, we can recover the queue from the log file, so the queue file is flushed asynchronously.
log file and queue file model
Consumption location file: The consumption location file is mainly used to store an offset of the queue consumed by different subscribers for a topic.
The following figure briefly describes the flow of messages on the server:
3. How to recover from disasters?
Speaking of disaster recovery, we talked about that every message will land on the broker in the previous section, but this is far from enough. We must ensure that the data can be recovered in the event of a single machine failure or even a failure of the computer room, so we need a set of A complete solution for data disaster recovery. In AMQ, a master-slave scheme is adopted, the master-slave is distributed in a data center, and the master-slave synchronous replication is such a scheme.
In JMQ, we have implemented a new set of replication solutions: one master-slave, at least one backup, master-slave distributed in the same data center, backup distributed in other data centers, master-slave synchronous replication, backup asynchronous replication such a way Perform data disaster recovery. Master-slave synchronous replication thus ensures that we have at least two full backups of the same message, and even if one is lost, we have another available. The backup node ensures that even if the entire data center fails in extreme cases, most of our messages can be recovered;
JMQ replication model
4. Push or pull? How are messages routed?
JMQ adopts the pull mode, that is, after the message is sent by the producer to the broker, the consumer actively initiates a request to fetch the message from the broker.
The routing of the sender is basically the same as the strategy adopted by the AMQ client.
Consumer routing and AMQ are almost random, the difference is that JMQ is a pull mode, and the broker adopts a long polling strategy.
5. How to deal with the message of consumption failure?
Different from AMQ, because the broker of JMQ directly supports retry, when the consumer fails to process the message, it directly sends a retry message command to the server, and the server receives the command and stores the message in the library. Then, when the consumer initiates a pull message command, the server takes out the message from the library according to a certain policy and returns it to the consumer for processing. One benefit of doing this is that the JMQ client reduces a third-party service dependency.
6. How to record message traces?
The overall process of the JMQ message track function is basically the same as that of AMQ. The main difference is that JMQ stores the message track related information in HBase, so that the storage period of the JMQ message track information becomes longer, and the amount that can be stored is larger. After the optimization of this method, the performance has also been greatly improved.
7. How to manage metadata?
From the first part, we know that the metadata of AMQ will be persisted in MySQL, and written to ZooKeeper at the same time of storage, and then distributed by ZK to the Broker and the client. So there are two problems:
The client introduces ZooKeeper, a third-party service. The more services are introduced, the worse the stability and maintainability of the client.
The registry, that is, ZooKeeper, is directly exposed to the client, which will lead to more and more connections in the registry, and lack of necessary means to protect the registry in the event of a failure.
In JMQ, we still use MySQL to persist metadata, and also write metadata to ZooKeeper, and ZooKeeper notifies the Broker, but the client no longer connects directly to ZooKeeper, but instead connects to the Broker to obtain metadata from the Broker. Data information. Since each Broker has a full amount of metadata information, the client can obtain metadata information by connecting to any Broker. This design brings several benefits:
The client's dependence on the ZooKeeper service is reduced. So far, our client only needs to communicate with the broker, the client logic has been simplified, and the stability of the client has been greatly improved.
ZooKeeper is no longer exposed to clients, so the stability of ZooKeeper is also guaranteed.
Since metadata can be obtained by connecting to any broker, even if a single broker is down in extreme cases, it will not affect the client's ability to obtain metadata, so from another point of view, this improves the availability of our registry.
8. Others
In addition to some basic problems mentioned above, we have also solved many problems, which are not listed here due to space problems, including but not limited to:
How to implement strictly sequential messages;
How to implement broadcast messages;
How to implement a two-phase transaction;
How to implement message playback.
The following is a rough overall architecture diagram of JMQ:
JMQ performance data
Test the machine used:
Scenario 1: One master and one slave, Master synchronously flushes disk + synchronously replicates to Slave
Scenario 2: One master and one slave, the Master asynchronously refreshes the disk + asynchronously replicates the Slave
JMQ scale
Number of Topic: 1000+
Number of access applications: 2500+
The number of messages entering the team in a single day: 50 billion+
JMQ3.0
By the middle of 2016, after two major versions of JMQ, the current version of JMQ 2.0 running online has been very stable and has excellent performance. Some people may ask, since everything is fine, do you have nothing to do? It just needs to be well maintained. In fact, it is not. With the development of the business and the expansion of the scale of the server, it is particularly important to build a diversified and automated system. Therefore, in the second half of this year, we have planned our JMQ3.0. We have several JMQ3.0s. Important goals:
Optimize the JMQ protocol;
Optimized replication model;
Achieve KAFKA protocol compatibility;
Realize global load balancing;
Implement a new electoral scheme;
Implement flexible scheduling of resources.
From these goals, it is not difficult to see that JMQ3.0 will have a major upgrade in function and architecture, so we intend to gradually achieve these goals through two major iterations. At present, we are in the process of joint debugging and testing of the first version of JMQ3.0, and it is expected that the first version will be pre-released at the end of December.
In the first version, we basically realized Kafka protocol compatibility, Raft-like master-slave election, global load balancing, optimized replication, and a simple demo with elastic scheduling. From some of our current test data, the new version has a certain improvement in performance. It is hoped that after the first phase of JMQ3.0 is launched, we will be able to take JMQ to a higher level through one or two iterations.
The following figure is an overall architecture of our JMQ3.0:
============Related Reading==============
Service-oriented framework technology selection and Jingdong JSF decryption
JD.com's road to JIMDB construction
This article is reproduced from IPD, IPD-Chat is the official public account of the basic platform department of Jingdong Mall.
The basic platform department includes technical directions such as databases, container clusters, middleware, image storage, application architecture, machine learning, and system assurance. Operates tens of thousands of servers in multiple data centers, supporting countless online tasks on JD.com. The IPD-Chat official account will share the current technical practice plan and latest project dynamics of the basic platform, pure technical dry goods, and seek breakthroughs together~ Welcome everyone to pay more attention and communicate together!