Detailed explanation of ActiveMQ message transmission mechanism and ACK mechanism
AcitveMQ is a message storage and distribution component, which involves all aspects of data interaction between client and broker. It not only guarantees the storage security of messages, but also provides additional means to ensure message security. Distribution is reliable.
1. ActiveMQ messaging mechanism
Producer client uses to send messages, Consumer client uses to consume messages; their collaboration center is ActiveMQ broker, broker is also a tool to decouple the calling process of producer and consumer, and finally realizes asynchronous RPC/ The function of data exchange. With the continuous development of ActiveMQ, it supports more and more features, and also solves the needs of developers to use ActiveMQ in various scenarios. For example, the producer supports asynchronous calls; the flow control mechanism is used to allow the broker to coordinate the consumption rate of the consumer; the consumer side can use prefetchACK to maximize the rate of message consumption; provide a "retransmission strategy" to improve message security, etc. We do not go into details here.
The life cycle of a message is as follows:
How does Samsung's Win8 tablet have a product now? Isn't it just released?
The picture simply describes the life cycle of a message, but in different architecture environments, the flow of messages may be more complicated. It will be explained in detail in the architecture of the broker later. After a message is sent from the producer side, once it is correctly saved by the broker, it will be consumed by the consumer, and then ACKed, and the broker side will delete it; however, when the message expires or the storage device When it overflows, it will also be terminated.
How does Samsung's Win8 tablet have a product now, isn't it just released?
This is a very complicated and messy picture; this picture briefly describes: 1) how the producer side sends messages 2) how the consumer side consumes messages 3) how the broker side schedules. If you use words to describe the concepts in the diagrams, I'm afraid it's hard to say. In the figure, prefetchAck is mentioned, as well as the basic logic of synchronous and asynchronous message sending; this will be of great help for you to understand the ACK mechanism below.
2. optimizeACK
"Optimizable ACK", this is ActiveMQ's optimization option for message ACK when the consumer consumes messages, and it is also one of the most important optimization parameters on the consumer side. You can enable it in the following ways:
1) In brokerUrl Add the following query string:
[java] view plain copy
String brokerUrl = "tcp://localhost:61616?" +
"jms.optimizeAcknowledge=true" +
"&jms.optimizeAcknowledgeTimeOut=30000" +
"&jms.redeliveryPolicy.maximumRedeliveries=6 ";
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(brokerUrl);
2) In destinationUri, add the following query string:
[java] view plain copy View snippet on CODE derived to my snippet
String queueName = "test-queue?customer.prefetchSize";
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination queue = session. createQueue(queueName);
We need to specify the optimizeACK option in brokerUrl and the prefetchSize option in destinationUri, where the brokerUrl parameter option is global, that is, all connections/sessions/consumers under the current factory will use these values by default; and The options in destinationUri are only valid in the consumer instance that uses this destination; if specified at the same time, the parameter option values in brokerUrl will be overwritten. optimizeAck indicates whether to enable "optimized ACK". Only in the case of true, the parameters of prefetchSize (hereinafter abbreviated as prefetch) and optimizeAcknowledgeTimeout will be meaningful. It should be noted here that the "optimizeAcknowledgeTimeout" option can only be configured in brokerUrl.
It is recommended to specify the prefetch value in destinationUri, because it is cumbersome to specify in brokerUrl; in brokerUrl, both queuePrefetchSize and topicPrefetchSize need to be set separately: "&jms.prefetchPolicy.queuePrefetch=12&jms.prefetchPolicy.topicPrefetch=12" and so on to specify one by one.
If prefetchACK is true, then prefetch must be greater than 0; when prefetchACK is false, you can specify prefetch as 0 and a positive number of any size. However, when prefetch=0, it means that the consumer will use the PULL method to obtain messages from the broker, and the broker will not actively push the message to the client until the client sends a PullCommand; when prefetch>0, The broker push mode is turned on. After that, as long as the client consumes and ACKs a certain message, it will immediately push multiple messages to the client.
When the consumer side uses the receive() method to obtain messages synchronously, prefetch can be 0 and any positive value; when prefetch=0, the receive() method will first send a PULL instruction and block until the broker side returns a message, This also means that messages can only be obtained one by one (similar to Request<->Response), which is also the PULL message mode in Activemq; when prefetch > 0, the broker will push a certain number of messages to the client in batches (<= prefetch) ,
When the consumer side uses MessageListener to obtain messages asynchronously, this requires that the prefetch value set by the development must be >=1, that is, at least 1; in the asynchronous consumption message mode, setting prefetch=0 is contrary, and a prefetch value will also be obtained. Exception.
In addition, we can also configure the "redelivery" strategy in brokerUrl, such as the maximum number of times that the broker can retransmit when a message is processed abnormally; it cooperates with the REDELIVERED_ACK_TYPE mentioned below. When a message needs to be retransmitted by the broker, the consumer will first delete it in the local "deliveredMessage queue" (a message queue that the consumer has received but not yet confirmed), and then send a confirmation command of type "REDELIVERED_ACK_TYPE" to the broker, and the broker will send the message to the broker. The message specified in the instruction is re-added to the pendingQueue (the message queue that needs to be sent to the consumer) until the right time, and pushes it to the client again.
By now, maybe you know the general meaning of optimizeACK and preference, but we may still have some doubts! ! The combination of optimizeACK and prefetch will achieve an efficient message consumption model: fetching messages in batches and "delaying" acknowledgments (ACK). Prefetch expresses the semantics of "bulk fetching" messages. The broker actively pushes multiple messages to the client in batches, which is much better than the way in which the client sends the PULL command multiple times and then the broker returns a message. It not only reduces the client's The number of blocking times and blocking time when acquiring messages can also greatly reduce network overhead. optimizeACK expresses the semantics of "delayed acknowledgment" (ACK timing). The client does not send an ACK after consuming a message, but caches it (pendingACK). When the number of these messages reaches a certain threshold, it only needs to pass A single ACK instruction acknowledges them all; this is much more performant than acknowledging each message individually. It can be seen that prefetch optimizes the performance of message transmission, and optimizeACK optimizes the performance of message acknowledgment.
When the rate of message consumption on the consumer side is high (relative to the producer producing messages), and the number of messages is also large (such as the continuous production of messages), we use optimizeACK + prefetch to greatly improve the performance of the consumer. But the other way around:
1) If the consumption speed on the consumer side is very slow (the processing of messages is time-consuming), an excessively large prefetchSize cannot effectively improve performance, but is not conducive to the load balancing on the consumer side (only for queues); according to good design guidelines , when the consumption speed of the consumer is very slow, we usually deploy multiple consumer clients, use a smaller prefetch, and close the optimizeACK at the same time, so that the message can be "load balanced" among multiple consumers (that is, evenly sent to each consumer) consumer); if the prefetchSize is larger, it will cause the broker to push a large number of messages to the client at one time, but these messages take a long time to ACK (message backlog), and when the client fails, it will also cause these messages to be resent.
2) If the consumer side consumes fast, but the producer side generates messages at a slower rate, for example, the producer generates 10 messages in 10 seconds, but the consumer can consume it in one second, and we have deployed multiple consumers! ! In this scenario, it is recommended to turn on optimizeACK, but you need to set a smaller prefetchSize; this ensures that each consumer can have "active work", otherwise one consumer will be very busy, but other consumers will hardly receive messages.
3) If the message is very important, especially if the "redelivery" message is received for no reason, then we need to optimizeACK=false, prefetchSize=1
Since optimizeACK is a "delayed" confirmation, then a potential risk is introduced: when the message is When there is no confirmation after consumption, the client side fails, then these messages may be resent to other consumers, so this risk requires the client side to be able to tolerate "duplicate" messages.
The prefetch value defaults to 1000. Of course, this value may be too large in many scenarios; we will not consider ACK_MODE (see below) for the time being. Usually, we only need to simply count the maximum number of messages consumed per second by a single consumer, namely Yes, for example, a consumer can process 100 messages per second, and we expect the consumer side to confirm every 2 seconds, then our prefetchSize can be set to 100 * 2 /0.65 is about 300. No matter how this value is set, the maximum number of messages held by the client is: prefetch + "DELIVERED_ACK_TYPE number of messages" (DELIVERED_ACK_TYPE see below)
even when optimizeACK is true, it will only take effect when the session's ACK_MODE is AUTO_ACKNOWLEDGE, That is, the consumer side will still not "delay acknowledgment" when other types of ACK_MODE are used, ie:
[java] view plain copy View code slice on CODE derived to my code slice
consumer.optimizeAck = connection.optimizeACK && session.isAutoAcknowledge()
When consumer.optimizeACK is valid, if the message (deliveredMessage) that the client has consumed but has not yet acknowledged reaches prefetch * 0.65, the consumer side will automatically ACK; at the same time, if the time interval from the last ACK has exceeded "optimizeAcknowledgeTimout" milliseconds, Also causes an automatic ACK.
In addition, simply add that when confirming messages in batches, you only need to specify "firstMessageId" and "lastMessageId" in the ACK command, that is, the message interval, then the broker side knows which messages the consumer (identified according to the consumerId) needs to confirm.
3. Introduction of ACK modes and types
The JMS API stipulates that the client can use four ACK_MODEs. In the javax.jms.Session interface:
AUTO_ACKNOWLEDGE = 1 Automatic confirmation
CLIENT_ACKNOWLEDGE = 2 Client manual confirmation
DUPS_OK_ACKNOWLEDGE = 3 Automatic batch confirmation
SESSION_TRANSACTED = 0 Transaction commit and confirmation
In addition, AcitveMQ adds a custom ACK_MODE:
INDIVIDUAL_ACKNOWLEDGE = 4 A single message confirms that
we will often use the above ACK_MODE when developing JMS applications, of which "INDIVIDUAL_ACKNOWLEDGE" is only supported by ActiveMQ, and of course developers can also use it It. ACK_MODE describes the way (timing) that the consumer and the broker confirm the message, such as when the consumer will confirm the message after the message is received by the consumer. For the broker, only when the ACK command is received will the message be considered correctly received or processed successfully. Through the ACK, a simple "guarantee" mechanism can be established between the consumer and the broker.
The client side specifies ACK_MODE, but when the client and the broker exchange the ACK command, they also need to inform the ACK_TYPE. ACK_TYPE indicates the type of the confirmation command. Different ACK_TYPE will convey the status of the message, and the broker can process the message according to different ACK_TYPE. different operations.
For example, when an exception occurs when the consumer consumes a message, it needs to send an ACK command to the broker, and the ACK_TYPE is "REDELIVERED_ACK_TYPE", then the broker will resend the message. ACT_TYPE is not defined in the JMS API because it is usually an internal mechanism and not intended for developers. The following ACK_TYPEs are defined in ActiveMQ (see the MessageAck class):
DELIVERED_ACK_TYPE = 0 Message "received", but not yet processed
STANDARD_ACK_TYPE = 2 "Standard" type, usually means the message "Processed successfully", the broker can delete the message
POSION_ACK_TYPE = 1 The message is "error", which usually means "abandoning" the message. For example, if the message cannot be processed correctly after being resent many times, the message will be deleted or DLQ (dead letter queue)
REDELIVERED_ACK_TYPE = 3 The message needs to be "retransmitted" ", for example, the consumer throws an exception when processing the message, and the broker will resend the message later.
INDIVIDUAL_ACK_TYPE = 4 means that only "single message" is confirmed, regardless of any ACK_MODE under
UNMATCHED_ACK_TYPE = 5, when forwarding messages between BROKERs, the receiver "rejects"
So far, we have understood the general principle: when the client side is in different ACK_MODE, it will mean sending ACK commands at different times, and each ACK command will contain ACK_TYPE, then the broker side can decide this according to ACK_TYPE. Follow-up operation of the message. Next, we analyze ACK_MODE and ACK_TYPE in detail.
[java] view plain copy View the code slice on CODE derived to my code slice
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
we need Specify ACK_MODE when creating a session. It can be seen that ACK_MODE will be shared by sessions, which means that all consumers under a session use the same ACK_MODE. When creating a session, the developer cannot specify other values than the ACK_MODE list. If the session is a transaction type, the user-specified ACK_MODE will be ignored, and the "SESSION_TRANSACTED" type will be used; if the session is not a transaction type, it will also be You can't set ACK_MODE to "SESSION_TRANSACTED", after all, this is contradictory.
How come Samsung's Win8 tablet has a product now, isn't it just released?
There are two styles of consumer message consumption: synchronous/asynchronous. ) is synchronous, and using messageListener is asynchronous; in the same consumer, we cannot use these two styles. For example, in the case of using listener, when the receive() method is called, an Exception will be obtained. In the two styles, the message confirmation timing is different.
"Synchronize"
[java] view plain copy View code slice on CODE derived to my code slice
//receive pseudocode---process
Message message = sessionMessageQueue.dequeue();
if(message != null){
ack(message);
} When the
return message
is called synchronously, ACK is called before the message returns from the receive method; therefore, if the client does not process it successfully, the message will be lost (possibly retransmitted, related to ACK_MODE).
"Asynchronous" pseudocode:
[java] view plain copy view snippet on CODE derived to my snippet
// based on listener
Session session = connection.getSession(consumerId);
sessionQueueBuffer.enqueue(message);
Runnable runnable = new Ruannale (){
run(){
Consumer consumer = session.getConsumer(consumerId);
Message md = sessionQueueBuffer.dequeue();
try{
consumer.messageListener.onMessage(md);
ack(md);//
}catch(Exception e){
redelivery();//sometime, not all the time;
}
}
//The thread pool will be used in the session to distribute Asynchronous message
//So multiple consumers in the same session can consume
threadPool.execute(runnable) in parallel;
when based on asynchronous calls, the confirmation of the message is after the onMessage method returns. If the onMessage method is abnormal, the message will be resent.
4. ACK_MODE is explained in detail
AUTO_ACKNOWLEDGE : automatic confirmation, which means that the confirmation timing of the message will be confirmed by the consumer. "Selective confirmation" seems to be full of uncertainty, which also means that developers must clearly know the specifics of "selective confirmation" Timing, otherwise it may lead to loss of messages, or repeated acceptance of messages. So in ActiveMQ, how does AUTO_ACKNOWLEDGE work?
1) For consumers, the optimizeAcknowledge property is only valid in AUTO_ACK mode.
2) Among them, DUPS_ACKNOWLEGE is also a potential AUTO_ACK, but the number and time of confirmation messages are different.
3) Before the "receive" method returns the message, it will check whether the optimizeACK option is turned on. If it is not turned on, the single message will be confirmed immediately, so in this case, after the message returns, if the developer is processing the message process If there is an exception in the message, the message will not be delivered, that is, "potential message loss"; if optimizeACK is enabled, it will be confirmed when the number of unAck reaches prefetch * 0.65, of course, we can specify prefetchSize = 1 to achieve message-by-message confirmation .
4) In the "asynchronous" (messageListener) mode, listener.onMessage(message) will be called first, and then ACK will be returned. If the onMessage method is abnormal, the client will send an additional confirmation command whose ACK_TYPE is REDELIVERED_ACK_TYPE; if the onMessage method is normal, The message will be acknowledged normally (STANDARD_ACK_TYPE). In addition, it should be noted that the number of retransmissions of a message is limited. Each message will contain a "redeliveryCounter" counter, which is used to indicate the number of times the message has been retransmitted. If the number of retransmissions reaches the threshold, a ACK_TYPE is the POSION_ACK_TYPE confirmation command, which causes the broker to think that the message cannot be consumed, and the message will be deleted or migrated to the "dead letter" channel.
Therefore, when we use the messageListener method to consume messages, it is usually recommended to use try-catch in the onMessage method, so that some information can be recorded when there is an error in processing the message, instead of letting the consumer continue to resend the message; if you do not use try-catch, There may be a problem of repeated reception of messages due to exceptions, You need to pay attention to whether the logic in your onMessage method is compatible with the judgment of duplicate messages.
How does Samsung's Win8 tablet have a product now, isn't it just released?
CLIENT_ACKNOWLEDGE : The client manually confirms, which means that AcitveMQ will not "self-claim" to ACK any message for you, and the developer needs to confirm at his own time. In this mode, developers need to pay attention to several methods: 1) message.acknowledge(), 2) ActiveMQMessageConsumer.acknowledege(), 3) ActiveMQSession.acknowledge(); 1) and 3) are equivalent and will All unacknowledged messages in the consumer in the current session are confirmed together. 2) Only those unacknowledged messages in the current consumer are confirmed. The developer may have to call the above method once at the right time.
We usually use CLIENT_ACKNOWLEDGE based on Group (message grouping), we will confirm the message (group) after a group's message sequence is accepted; however, when you think the message is important, it can only be confirmed when the message is processed correctly It is also possible to use this ACK_MODE when
If the developer forgets to call the acknowledge method, it will cause duplicate messages to be received when the consumer restarts, because for the broker, those messages that have not been actually ACKed are considered "unconsumed".
Developers can immediately call the message.acknowledge() method to confirm messages "one by one" after the current message is successfully processed, which can minimize the number of message retransmissions due to network failures; of course, multiple messages can also be processed After that, the acknowledge method is called intermittently to acknowledge multiple messages at a time, reducing the number of acks to improve the efficiency of the consumer, but this is still a trade-off of pros and cons.
In addition to the message.acknowledge() method, ActiveMQMessageConumser.acknowledge() and ActiveMQSession.acknowledge() can also confirm messages, but the former only confirms the messages in the current consumer. where sesson.acknowledge() and message.acknowledge() are equivalent.
ActiveMQ will not send STANDARD_ACK_TYPE until message.acknowledge() is called, regardless of "sync"/"async". If the number of unacknowledged messages on the client side reaches prefetchSize * 0.5, an additional confirmation command with ACK_TYPE of DELIVERED_ACK_TYPE will be sent, which will trigger the broker side to continue to push messages to the client side. (See PrefetchSubscription.acknwoledge method)
On the broker side, for each Consumer, a number of "delayed" messages due to "DELIVERED_ACK_TYPE" will be saved. This parameter is prefetchExtension. In fact, this value will not be greater than prefetchSize * 0.5, because the Consumer side will strictly control the sending of the DELIVERED_ACK_TYPE command. The timing (see ActiveMQMessageConsumer.ackLater method), the broker side cooperates with "prefetchExtension" and prefetchSize to determine the number of messages to be pushed to the client side, count = prefetchExtension + prefetchSize - dispatched.size(), where dispatched indicates that it has been sent to the client The client side does not yet have the total amount of "STANDARD_ACK_TYPE" messages; it can be seen that in the CLIENT_ACK mode, calling the acknowledge() method quickly enough is to determine the rate at which the consumer side consumes messages; if the client side for some reason causes the acknowledge method to fail If it is executed, a large number of messages will not be confirmed, and the broker will not push messages. In fact, the client will be in a state of "suspended death" and cannot continue to consume messages. We require the client to acknowledge() once before consuming 1.5*prefetchSize messages; usually we always call it once for every message consumed, which is a good design.
In addition, it needs to be added: all ACK commands are sent to the broker side in turn. In CLIET_ACK mode, before the message is delivered to the listener, an ACK command of DELIVERED_ACK_TYPE will be created first, until the unacknowledged message on the client side reaches "prefetchSize * This ACK command is sent only when it is 0.5". If the developer calls the acknowledge() method before that, the message will be directly acknowledged (STANDARD_ACK_TYPE). The broker side usually considers the "DELIVERED_ACK_TYPE" confirmation command as a "slow consumer" signal. If the consumer cannot acknowledge the message in time and the broker side is blocked, the consumer will be marked as "slow". The message will be forwarded to other Consumers.
DUPS_OK_ACKNOWLEDGE : "message can be repeated" confirmation, which means that in this mode, there may be repeated messages, not a message needs to send multiple ACKs. It is a potential "AUTO_ACK" acknowledgment mechanism, born for batch acknowledgment, and has the characteristics of "delayed" acknowledgment. For developers, the code structure in this mode is the same as AUTO_ACKNOWLEDGE, and there is no need to call the acknowledge() method to acknowledge the message like CLIENT_ACKNOWLEDGE.
1) In ActiveMQ, if the Destination is a Queue channel, we can really think that DUPS_OK_ACK is the case of "AUTO_ACK + optimizeACK + (prefetch > 0)", and the confirmation timing is almost the same; in addition, in this mode, if If prefetchSize = 1 or optimizeACK is not enabled, it will also cause messages to be confirmed one by one, thus losing the feature of batch confirmation.
2) If the Destination is Topic, DUPS_OK_ACKNOWLEDGE will have the meaning explained in the JMS specification, that is, regardless of whether optimizeACK is enabled, it will be confirmed in batches (STANDARD_ACK_TYPE) when the number of messages consumed is >=prefetch * 0.5. A confirmation command of DELIVERED_ACK_TYPE will be sent, which is the biggest difference between 1) and AUTO_ACK.
This also means that when the consumer fails to restart, those messages that have not been ACKed will be resent.
SESSION_TRANSACTED : This mode is used when the session uses transactions. After the transaction is opened, and before session.commit(), all consumed messages are either confirmed normally or all redelivery. This rigor is usually especially suitable in GROUP-based (message grouping) or other scenarios. In SESSION_TRANSACTED mode, optimizeACK has no effect, because in this mode, optimizeACK will be forced to false, but prefetch can still determine the timing of sending DELIVERED_ACK_TYPE.
Because the Session is not thread-safe, all consumers in the current session will share the same transactionContext; at the same time, it is recommended that there is only one Consumer in a transaction-type Session to avoid the rollback() or commit() method being called by multiple consumers. The news is confusing.
When the consumer receives the message, it first checks whether the TransactionContext has been opened. If not, it will open and generate a new transactionId, and send the information to the broker; after that, it will check whether the number of messages that have been consumed in the transaction is >= prefetch * 0.5 , If it is greater than, send a "DELIVERED_ACK_TYPE" confirmation command; then start to call the onMessage() method, if it is synchronous (receive), then return the message. There is nothing special about the above process, and other confirmation modes.
When the developer decides that the transaction can be committed, the session.commit() method must be called. The commit method will cause all messages in the transaction of the current session to be confirmed immediately; during the confirmation process of the transaction, the unconfirmed messages in the local deliveredMessage queue are firstly All messages are confirmed (STANDARD_ACK_TYPE); then send the transaction submission command to the broker and wait for the broker's feedback. If the broker-side transaction operation is successful, the local deliveredMessage queue will be emptied and a new transaction will begin; if the broker-side transaction operation fails (at this time the broker already rolledback), then for the session, inner-rollback will be executed. What this rollback does is to clear the message in the current transaction and ask the broker to resend (REDELIVERED_ACK_TYPE), and the commit method will throw an exception.
When the session.commit method is abnormal, it is usually for developers to call session.rollback() to roll back the transaction (in fact, there is no problem if the developer does not call it). Of course, you can call rollback() at any time after the transaction starts. ,rollback means the end of the current transaction, all messages in the transaction will be resent. It should be noted that whether it is inner-rollback or calling session.rollback() to cause message retransmission, it will cause the message.redeliveryCounter counter to increase, which will eventually be limited by the "jms.redeliveryPolicy.maximumRedeliveries" configured in brokerUrl, if the number of rollbacks If there are too many, and the upper limit of the number of retransmissions is reached, the message will be DLQ (dead letter).
INDIVIDUAL_ACKNOWLEDGE : Single message confirmation, this confirmation mode is rarely used, and its confirmation timing is almost the same as CLIENT_ACKNOWLEDGE. When the message is successfully consumed, you need to call message.acknowledege to confirm the message (single), and CLIENT_ACKNOWLEDGE mode first message. The acknowledge() method will cause all messages in the entire session to be acknowledged (batch acknowledgment).
Conclusion: So far, we have briefly understood the message transmission mechanism in ActiveMQ, as well as the ACK strategy in JMS, focusing on the optimizationACK strategy, hoping that developers can avoid some unnecessary errors in using activeMQ. If there are any omissions and mistakes in this article, please feel free to let me know, thank you very much.
Source code reference class:
1) ActiveMQConnectionFactory, ActiveMQMessageConsumer, ActiveMQSession, MessageAck, etc.
2) Queue,PrefetchSubscription,TransactionContext,TransactionStore
摘自：http://blog.csdn.net/lulongzhou_llz/article/details/42270113
ActiveMQ messaging mechanism and ACK mechanism
Guess you like
Origin http://10.200.1.11:23101/article/api/json?id=326606880&siteId=291194637
Recommended
Ranking