Excerpt from "rocketmq practical principles and analysis"
Producers and consumers are two important roles message queue, the producer writes data to a message queue, the consumer reads the data from the message queue. Most Rocketmq users need only concern producers and consumers, this article will focus on the characteristics of producers and consumers and their offset and logs
Consumers can be divided into two different types, one is DefaultMQPushConsume, a read operation by the system control, automatic call processing method after receiving an incoming message, the other is DefaultMQPullConsume, the read operation of most of the functions are autonomous control by the user.
DefaultMqPushConsume use
push mode is such a good variety of main parameters and the incoming message function, the system will automatically call after receiving the message processing function to process the message, automatically save offset, and load balancing automatically added after the new DefaultMqPushConsume, as follows.
package rocketmq.day02;
import com.alibaba.rocketmq.client.consumer.DefaultMQPushConsumer;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import com.alibaba.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.common.consumer.ConsumeFromWhere;
import com.alibaba.rocketmq.common.protocol.heartbeat.MessageModel;
import java.io.UnsupportedEncodingException;
/**
* @author heian
* @create 2019-12-09-8:12 上午
* @description
*/
public class QuickStart {
public static void main(String[] args) throws MQClientException {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("unique_consume_group_name");
consumer.setNamesrvAddr("192.168.142.133:9876");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.setMessageModel(MessageModel.CLUSTERING);//默认是集群模式
consumer.subscribe("topicName",null);
consumer.registerMessageListener((MessageListenerConcurrently) (listMsg, consumeConcurrentlyContext) -> {
byte[] body = listMsg.get(0).getBody();
try {
String ms = new String(body,"utf-8");
System.out.println(Thread.currentThread().getName()+"收到消息：" + ms);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
});
}
}
DeafultMqPushConsume need to set the three parameters, the consumer group name consumeGroupName, address port NameServer, topic name Topic.
1) the name of the consumer group for a plurality of consumers consumer organizations together to improve concurrency, and generally used in conjunction with the message MessageModel mode, generally two types of patterns, and broadcasting Boradcasting set Clustering
2) cluster model the same below a consumer group name which consumers consume only part of the subscription message, the same consumer group name in consumer spending is the sum of subscription news overall content of the topic, so as to achieve load balanced
3) Broadcast mode the same below each consumer a consumer group name consumer can subscribe to all the news topic, that is, a message will be distributed several times, consumed more consumers
4) NameServer can fill a plurality of address port, separated by semicolons, thereby eliminating the single point of failure such as: ip1: port; ip2: port2
4) topic name used to indicate the message type, you need to create in advance. If you do not consume a topic of news content can be filtered according to tag, if consume.subsrcibe ( "topicName", "tag1 || tag2 || tag3"), indicate consumers only need to subscribe tag1 2 3 in this topic is news, or * null if the fill represents a subscription for all the messages of the topic
DefaultMqPushConsume process flow
DefaultMqPushConsume main function DefaultMqPushConsumerImpl class, message processing logic is PullCallback pullMessage this function callback class, this class has a switch statement, do deal with want to use according to the type of message broker to return, but you can see why the next push mode pullRequest use it? It is reached by a long push polling way, advantages of both Pull the long polling, there's real-time push.
PullCallback pullCallback = new PullCallback() {
@Override
public void onSuccess(PullResult pullResult) {
if (pullResult != null) {
pullResult =
DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(
pullRequest.getMessageQueue(), pullResult, subscriptionData);
switch (pullResult.getPullStatus()) {
case FOUND:
long prevRequestOffset = pullRequest.getNextOffset();
pullRequest.setNextOffset(pullResult.getNextBeginOffset());
long pullRT = System.currentTimeMillis() - beginTimestamp;
DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(
pullRequest.getConsumerGroup(), pullRequest.getMessageQueue().getTopic(), pullRT);
long firstMsgOffset = Long.MAX_VALUE;
if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
}
else {
firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();
DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(
pullRequest.getConsumerGroup(), pullRequest.getMessageQueue().getTopic(),
pullResult.getMsgFoundList().size());
boolean dispathToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(//
pullResult.getMsgFoundList(), //
processQueue, //
pullRequest.getMessageQueue(), //
dispathToConsume);
if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
}
else {
DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
}
}
if (pullResult.getNextBeginOffset() < prevRequestOffset//
|| firstMsgOffset < prevRequestOffset) {
log.warn(
"[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",//
pullResult.getNextBeginOffset(),//
firstMsgOffset,//
prevRequestOffset);
}
break;
case NO_NEW_MSG:
pullRequest.setNextOffset(pullResult.getNextBeginOffset());
DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);
DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
break;
case NO_MATCHED_MSG:
pullRequest.setNextOffset(pullResult.getNextBeginOffset());
DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);
DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
break;
case OFFSET_ILLEGAL:
log.warn("the pull request offset illegal, {} {}",//
pullRequest.toString(), pullResult.toString());
pullRequest.setNextOffset(pullResult.getNextBeginOffset());
pullRequest.getProcessQueue().setDropped(true);
DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {
@Override
public void run() {
try {
DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(
pullRequest.getMessageQueue(), pullRequest.getNextOffset(), false);
DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest
.getMessageQueue());
DefaultMQPushConsumerImpl.this.rebalanceImpl
.removeProcessQueue(pullRequest.getMessageQueue());
log.warn("fix the pull request offset, {}", pullRequest);
}
catch (Throwable e) {
log.error("executeTaskLater Exception", e);
}
}
}, 10000);
break;
default:
break;
}
}
}
@Override
public void onException(Throwable e) {
if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
log.warn("execute the pull request exception", e);
}
DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
PullTimeDelayMillsWhenException);
}
}Push mode is terminated after the server receives the message, the message is pushed to the active client to real-time high. For a queue for providing server-side service, the use of push push way has many disadvantages.
- Increase the workload of the server side, thereby affecting the performance of server-side
- client-side processing capabilities vary, from server-side client end state control, if the client-side can not process the message server1 pushed over the presence of various potential problems
Pull embodiment is client-side circular drive away the message from the server, the client-side initiative in the hands that he got me to a certain message, then taken in properly processed. Pull approach is the problem:
- Got me bad news cycle interval setting, the interval is too short to be in a state of "busy wait", a waste of resources.
- Pull each time interval is too long, time to end the message server, there may not be timely manner.
Long polling manner by mating client and server side, to pull both have advantages, but also achieve the purpose of ensuring real time property. In this period, he also impl source, which request.setSuspendTimeOutMills (brokerSuspendMaxTimeMills), set the broker role is the longest blocking time, the default is 15s, note that this is a broker in the absence of new information will be blocked, the news will be returned immediately.
private static final long BrokerSuspendMaxTimeMillis = 1000 * 15;
(Not shown) of the source can be seen from the broker, the server receives the request, if no new message queue is not eager to return, but continue to view the status of a loop through each waitForRunning period (the default five seconds ), then check. By default, when the broker has been no new message, the third check, the wait time exceeds the request inside brokerSuspendMaxTimeMills, it returns an empty result. Directly call notifyMessageArriving function returns the result of the request after waiting for the process, broker received a new message. Core "long polling" is the broker client side hold live request over a short time, a new message arrives, it returns the message using the existing connection to consume immediately at this time. "Long Polling" initiative still rests in the end consume, broker timely a large backlog of messages will not be actively pushed to consume.
Limitations long polling request is hold live consume resources occupied when needed, it is suitable for a controlled amount of this client message queue scene. Then we can see whether it is push mode or pull mode, are using pullRequest way (long polling), just push mode is the method server to help clients achieve pull message from the broker in the (broker has continually polls immediately return a message, no message is circulating constantly check the status of the server at intervals of 5 seconds so the cycle three times, that is, 15 seconds still no news, an empty result is returned), and pull their own way the user is required to achieve.
Flow control DefaultMqPushConsume
This section analyzes the flow control method of push, PushConsume the core or pull method, so that in this way the client can be adjusted according to the speed of the acquired message to its processing speed. Because it is multi-threaded approach to achieve, flow control is much more complex than single-threaded. (Class ConsumeMessageOrderlyService implements ConsumeMessageService)
this.consumeExecutor = new ThreadPoolExecutor(//
this.defaultMQPushConsumer.getConsumeThreadMin(),//
this.defaultMQPushConsumer.getConsumeThreadMax(),//
1000 * 60,//
TimeUnit.MILLISECONDS,//
this.consumeRequestQueue,//
new ThreadFactoryImpl("ConsumeMessageThread_"));//修改线程后缀名Pull gets the message if submitted directly to the thread pool is difficult to monitor and control the execution, such as: how to get the message of the amount of accumulation? How to deal with repeat some of the messages? How to delay processing certain messages? rocketmq given a snapshot class processQueue to solve these problems, when this class is running, each message queue, there will be a corresponding processQueue the object, saved the message queue message processing snapshot of the state. (Processquene class member variables)
private final Logger log = ClientLogger.getLog();
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();
private volatile long queueOffsetMax = 0L;
private final AtomicLong msgCount = new AtomicLong();
.......The main content ProcessQueue target is actually stored in a treemap and readwritelock, the former in order to offset message queue as a key, to pass a value within the message saved all but the message has not been processed from the message queue to obtain; read-write lock control with concurrent access to multiple threads of treemap.
With processqueue objects, flow control is easy and a lot of flexibility, and client requests before each pull will do the following three judge to control the flow, as follows: Slightly
pushConsumer determines message has not acquired but the number of process (thread pool queue), the total size of the message, offset span, any value as long as it exceeds the size of the set at intervals in the pull message, so as to achieve the flow control purpose. In addition processQueue can also assist in achieving the logical order of consumption.
DefaultMQPullConsumer use
Use DefaultMQPullConsumer the same as using DefaultMqPushConsume need to set various parameters, write function processes the message, but need to do some extra things. Please take over with examples of source to introduce org.apache.rocketmq.example.simple package, the following code ( https://github.com/apache/rocketmq )
package rocketmq.day03;
import com.alibaba.rocketmq.client.consumer.DefaultMQPullConsumer;
import com.alibaba.rocketmq.client.consumer.PullResult;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.common.message.MessageQueue;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
/**
* @author heian
* @create 2019-12-26-12:47 下午
* @description pull消费者 地址：https://github.com/apache/rocketmq/blob/master/example/src/main/java/org/apache/rocketmq/example/simple/PushConsumer.java
*/
public class PullConsume {
//保存offset
private static final Map<MessageQueue, Long> OFFSE_TABLE = new HashMap<MessageQueue, Long>();
public static void main(String[] args) throws MQClientException {
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("pull_consume_group");
consumer.setNamesrvAddr("192.168.0.102:9876");// 192.168.0.102
consumer.start();
Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("topicName");
for (MessageQueue mq : mqs) {
System.out.println("当前队列"+mq);
SINGLE_MQ:
while (true) {
try {
long start = System.currentTimeMillis();
PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
long end = System.currentTimeMillis();
System.out.println("耗时：" + (end-start));
System.out.println("拉取结果："+pullResult);
putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
System.out.println("存储在内存："+OFFSE_TABLE);
switch (pullResult.getPullStatus()) {
case FOUND:
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break SINGLE_MQ;
case OFFSET_ILLEGAL:
break;
default:
break;
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
consumer.shutdown();
}
private static long getMessageQueueOffset(MessageQueue mq) {
Long offset = OFFSE_TABLE.get(mq);
if (offset != null)
return offset;
return 0;
}
private static void putMessageQueueOffset(MessageQueue mq, long offset) {
OFFSE_TABLE.put(mq, offset);
}
}
Before the message has been sent over, mqAdmin console messages as follows :( If your broker did not start end appears to be empty)
console output idea is as follows:
当前队列MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]
耗时：9
拉取结果：PullResult [pullStatus=FOUND, nextBeginOffset=8, minOffset=0, maxOffset=8, msgFoundList=8]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8}
耗时：20989
拉取结果：PullResult [pullStatus=NO_NEW_MSG, nextBeginOffset=8, minOffset=0, maxOffset=8, msgFoundList=0]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8}
当前队列MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=1]
耗时：4
拉取结果：PullResult [pullStatus=FOUND, nextBeginOffset=12, minOffset=0, maxOffset=12, msgFoundList=12]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=1]=12}
耗时：20005
拉取结果：PullResult [pullStatus=NO_NEW_MSG, nextBeginOffset=12, minOffset=0, maxOffset=12, msgFoundList=0]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=1]=12}
当前队列MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=2]
耗时：3
拉取结果：PullResult [pullStatus=FOUND, nextBeginOffset=8, minOffset=0, maxOffset=8, msgFoundList=8]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=1]=12, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=2]=8}
耗时：20007
拉取结果：PullResult [pullStatus=NO_NEW_MSG, nextBeginOffset=8, minOffset=0, maxOffset=8, msgFoundList=0]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=1]=12, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=2]=8}
当前队列MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=0]
耗时：4
拉取结果：PullResult [pullStatus=FOUND, nextBeginOffset=30, minOffset=0, maxOffset=30, msgFoundList=30]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=1]=12, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=2]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=0]=30}
耗时：25005
拉取结果：PullResult [pullStatus=NO_NEW_MSG, nextBeginOffset=30, minOffset=0, maxOffset=30, msgFoundList=0]
存储在内存：{MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=3]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=1]=12, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=2]=8, MessageQueue [topic=topicName, brokerName=humingmingdeMacBook-Pro.local, queueId=0]=30}
It can be found when polling each queue, there is no memory look the message queue is not saved to the first memory, and then the second save for polling message queue, the consumer pulling the end message broker like (there was no over the new message, the default source 10 seconds blocking private long consumerPullTimeoutMillis = 1000 * 10) , end of message broker is also hold for a while, then the message stored in the memory, please refer to analytical principle pull message: pull parsing source
public class DefaultMQPullConsumer member variables:
- brokerSuspendMaxTimeMillis 20s default maximum timeout length suspended in polling mode, if the return PULL_NOT_FOUND, not immediately returned to the client according to the time pull offset lookup message from a file stored in the Broker end, but to the PullRequestHoldService thread, every 5 seconds to go to pull a message, the message to pull the client if found return it, otherwise a timeout.
- consumerTimeoutMillisWhenSuspend: 30s default message whole pulling process, pulling the client waits for the server response result timeout
- consumerPullTimeoutMillis: establish network connection timeout default 10s pull news
- messageModel: consumption patterns, broadcast, cluster
- messageQueueListener ： 业务消息监听器
- OffsetStore ：消息消费进度管理器
- registerTopics ：注册主题数
- allocateMessageQueueStrategy ：队列分配器
- maxReconsumeTimes ：最大消息重试次数，默认16次
（1）获取MessagQueue 并遍历
一个Topic包含多个MessagQueue，如果这个consume需要获取topic所有的消息，就要遍历所有的MessagQueue，如果有特殊情况也可以选定某些特定的MessagQueue 来读取消息
（2）维护Offsetstore
从一个MessagQueue里拉取消息的时候，需要传入offset参数，随着不断的读取消息，offset会不断增长。这个时候由用户负责把offset存储下来，根据具体的情况可以存到内存或者写入到磁盘或者数据库等
（3）根据不同的消息状态做不同的消息处理
拉取消息的请求发出后，会返回 FOUND NO_MATCHED_MSG NO_NEW_MSG OFFSET_ILLEAGER四种状态，需要根根据每个状态做不同的消息处理，比较重要的两个状态是FOUND NO_NEW_MSG，分别表示获取到消息和没有新的消息
但我觉得这上面拉取消息是不合理的，因为你拉取消息后还得在拉取一遍，直到消息返回状态为无此消息才跳出循环拉取下一条。
实际情况中可以把while true循环放到外层，达到无限循环的目的，不断循环去拉去消息队列的消息（会有新的消息）因此pullConsume需要用户自己处理遍历MessagQueue，所以pullConsume有更多的自主性和灵活性。
consume的启动 关闭流程
消息队列一般是提供一个不间断的持续性服务，consume在使用过程中如何才能更优雅的启动和关闭，确保消息不漏掉和重复消费呢？
consume分为push和pull两种方式，对于pullConsume来说使用者的主动权很高，可以根据实际情况去暂停 启动 和停止消费过程。需要注意的是offset的保存，要在程序的异常处理部分增加把offset写入磁盘方面的处理，记住了每个MessagQueue的offset，才能保证消息的准确性。
pushConsume在启动的时候会做各种配置的检查，然后连接nameserver获取topic信息，在启动的时候如果遇到异常，比如无法连接nameserver，程序仍然可以正常启动不报错。（在单机环境下可以测试下），那为什么不直接报错退出呢？这个和分布式设计有关，RocketMq集群可以有对歌nameserver broker，某个机器出了问题整体服务依然可用，所以在DefaultMQPushConsumer被设计成当发现某个连接异常时不立刻退出，而是不断尝试重新连接。可以进行这样一个测试，在DefaultMQPushConsumer正常运行的时候，手动kill调broker或者name server 过一会在启动，会发现DefaultMQPushConsumer不会出错退出，在服务恢复后可正常运行，在服务不可用的期间仅仅是在日志里报异常信息（对应的broker或者name server日志）。
如果需要在消费者启动的时候去报露错误，可以在consume.start()启动后调用，如下，其会抛出MQClientException异常（配置信息不准确或者当前服务不可用）
Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("topicname");
不同类型的生产者
生产者向消息队列写数据，不同的业务场景需要生产者采取不同的策略。比如同步发送 异步发送 发送事物消息等，下面具体介绍。
package rocketmq.day01;
import com.alibaba.rocketmq.client.exception.MQBrokerException;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.client.producer.DefaultMQProducer;
import com.alibaba.rocketmq.client.producer.SendResult;
import com.alibaba.rocketmq.common.message.Message;
import com.alibaba.rocketmq.remoting.exception.RemotingException;
import java.text.SimpleDateFormat;
import java.util.Date;
public class ProducerDemo {
public static void main(String[] args) throws MQClientException {
DefaultMQProducer producer = new DefaultMQProducer("unique_producer_group__name");
producer.setNamesrvAddr("192.168.142.133:9876");
producer.start();
for (int i = 0; i < 1; i++) {
Date date = new Date();
SimpleDateFormat sdf = new SimpleDateFormat();
String format = sdf.format(date);
Message message = new Message("topicName", String.valueOf(i),format.getBytes());
SendResult sendResult= new SendResult();
try {
sendResult = producer.send(message);//有消息返回
} catch (RemotingException |MQBrokerException | InterruptedException e) {
System.out.println("消息发送失败：" + sendResult.getMsgId());
e.printStackTrace();
}
System.out.println("key:"+i + "消息的发送结果为：" + sendResult.toString() + "消息ID为：" + sendResult.getMsgId());
}
producer.shutdown();
}
}
发送消息经历五个步骤：
- 设置producer的GroupName
- 设置InstanceName 当一个jvm需要启动多个procuder的时候，通过设置不同的InstanceName来区分，不设置的话默认使用“DEFAULT"
- 设置发送失败重试次数，当网络出现异常的时候，这个次数影响消息的重复投递次数，想要保证消息不丢，可以设置多重试
- 设置nameserver 地址
- 组装并发送
消息的发送有同步发送和异步发送两种，上面使用的是异步。消息发送返回的状态有：FLUSH_DISK_TIMEOUT FLUSH_SLAVE_TIMEOUT SLAVE_NOT_AVAILABKE SEND_OK 不同状态在不同的刷盘策略和不同的同步策略的含义是不同的。
- FLUSH_DISK_TIMEOUT：表示没有在规定的时间内完成刷盘（需要broker的刷盘策略被设置成SYNC_FLUSH才会报这个错）
- FLUSH_SLAVE_TIMEOUT：表示在主备方式下，并且broker 被设置成SYNC_MASTER方式，没有在设定时间内完成主从同步
- SLAVE_NOT_AVAILABKE：这个状态产生的场景和FLUSH_SLAVE_TIMEOUT类似，表示主备方式下，并且broker被设置成SYNC_MASTER,但是没有找到被配置成slave的broker
- SEND_OK：表示发送成功，发送成功的具体含义，比如消息是否已经被存储到磁盘？消息是否被同步到slave上？消息在slave上是否被写入磁盘？需要结合所配置的刷盘策略 主从策略来定，这个状态还可以简单的理解为，没有发生上面列出的三个问题状态就是SEND_OK
写一个高质量的生产者程序，重点在于发送结果的处理，要充分考虑各种异常，写清楚对应的处理逻辑。
（1）发送延迟消息
支持发送延迟消息，broker接受到消息后，延迟一段时间在处理，使得消息在规定的一段时间内生效，延迟消息的使用方法是创建mmessage对象时，调用setDelayTimeLevel(int level) 方法设置，目前延迟消息不支持任意设置，仅支持如下（1s/5s/10s/30s/1m/2m/3m/4m/5m/6m/7m/8m/9m/10m/20m/30m/1h/2h)，比如setDelayTimeLevel(3)表示延迟10s。
message.setDelayTimeLevel(3);
(2) 自定义消息发送规则
一个topic会有多个message queue，如果使用procuder的默认配置，这个procuder会轮流像各个message queue发送消息。consume在消费消息的时候，会根据负载均衡策略，消息被分配到对应的message queue，如果不经过特定的设置，某条消息被发到哪个message queue，被哪个consume消费是未知的。
如果业务需要我们把消息发送到指定的消息队列，那该怎么办？
package rocketmq.day03;
import com.alibaba.rocketmq.client.exception.MQBrokerException;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.client.producer.DefaultMQProducer;
import com.alibaba.rocketmq.client.producer.MessageQueueSelector;
import com.alibaba.rocketmq.client.producer.SendResult;
import com.alibaba.rocketmq.common.message.Message;
import com.alibaba.rocketmq.common.message.MessageQueue;
import com.alibaba.rocketmq.remoting.exception.RemotingException;
import org.aspectj.weaver.ast.Var;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;
/**
* @author heian
* @create 2019-12-30-11:24 上午
* @description 指定生产者往某一个特定的队列中发消息
*/
public class MyMessageQueueSelect implements MessageQueueSelector {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object orderKey) {
int id = Integer.parseInt(orderKey.toString());
int idMainIndex = id/100;
int size = mqs.size();
int index = idMainIndex%size;
return mqs.get(index);
}
public static void main(String[] args) throws MQClientException {
DefaultMQProducer producer = new DefaultMQProducer("unique_producer_group__name");
MyMessageQueueSelect myMessageQueueSelect = new MyMessageQueueSelect();
producer.setRetryTimesWhenSendFailed(3);//重试次数
producer.setNamesrvAddr("192.168.142.133:9876");//多个用；分割
producer.start();
for (int i = 0; i < 1; i++) {
Date date = new Date();
SimpleDateFormat sdf = new SimpleDateFormat();
String format = sdf.format(date);
Message message = new Message("topicName", String.valueOf(i),format.getBytes());
SendResult sendResult= new SendResult();
try {
String orderKey = "";
sendResult = producer.send(message, myMessageQueueSelect, orderKey);
} catch (RemotingException | MQBrokerException | InterruptedException e) {
System.out.println("消息发送失败：" + sendResult.getMsgId());
e.printStackTrace();
}
System.out.println("key:"+i + "消息的发送结果为：" + sendResult.toString() + "消息ID为：" + sendResult.getMsgId());
}
}
}
发送消息的时候需要把MessageQueueSelector 的对象作为参数，使用public SendResult send(Message msg,MessageQueueSelector selector,Object arg)函数发送消息即可。在MessageQueueSelector的实现中，根据需要传入orderkey参数或者根据message消息内容确定把消息发到哪个message queue上，返回被选中的message queue
生产者源码
今天测试环境出现了一个bug,看报错信息如下com.alibaba.rocketmq.client.impl.producer.DefaultMQProducerImpl#sendDefaultImpl方法中的Validators.checkMessage(msg, this.defaultMQProducer);方法
if (msg.getBody().length > defaultMQProducer.getMaxMessageSize()) {
throw new MQClientException(ResponseCode.MESSAGE_ILLEGAL,
"the message body size over max value, MAX: " + defaultMQProducer.getMaxMessageSize());
}因为我的消息体是按照1000个发送的其大小在1兆多点，而使用平安内部框架观察源码最大限制是4兆，这就很奇怪为什么我消息体没超过限制却发生了这样的报错呢？com.alibaba.rocketmq.client.producer.DefaultMQProducer
private String createTopicKey = MixAll.DEFAULT_TOPIC;
private volatile int defaultTopicQueueNums = 4;
private int sendMsgTimeout = 3000;
private int compressMsgBodyOverHowmuch = 1024 * 4;
private int retryTimesWhenSendFailed = 2;
private boolean retryAnotherBrokerWhenNotStoreOK = false;
private int maxMessageSize = 1024 * 128; //平安的这里是1024*1024*4
private boolean unitMode = false;
而且看源码发现了一个比较有意思的语法，也是在com.alibaba.rocketmq.client.impl.producer.DefaultMQProducerImpl#sendDefaultImpl
while(true){
lable122:{
String info;
if(times<timesTotal){
....
}
break lable122；
}
times++;
}其实也很好理解，就是你在while里自定一个一个代码块，每次执行的时候你跳出的不是整个循环而是自己自定义的代码块，如果while中有多个代码块，二者是相互隔离的，如下：跳出的代码块不再执行
public static void main(String[] args) {
label1:
for (int i=0;i<=100;i++){
label2:
if (i>6){
System.out.println("label2="+i);
break label2;
}
if (i==10){
System.out.println("the end");
break label1;
}
label3:
if (i>8){
System.out.println("break label3");
break label3;
}
}
}label2=7
label2=8
label2=9
break label3
label2=10
the end
