behavioral triggers
The persistence of consumer consumption offset position is the behavior of the consumer client, which is a timed task set when the client starts, as follows:
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
MQClientInstance.this.persistAllConsumerOffset();
} catch (Exception e) {
log.error("ScheduledTask persistAllConsumerOffset exception", e);
}
}
}, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
The scheduled task is triggered with a delay of 10s, and then triggered once every specific time. If there is no special setting, the default time interval is 5s, as shown below:
/**
* Offset persistent interval for consumer
*/
private int persistConsumerOffsetInterval = 1000 * 5;
Persistence behavior
In the process of persistence, first obtain all consumers from the consumer registry of the current client, and then perform specific persistence operations one by one in an iterative manner. The source code is as follows:
private void persistAllConsumerOffset() {
Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQConsumerInner> entry = it.next();
MQConsumerInner impl = entry.getValue();
impl.persistConsumerOffset();
}
}
When performing a specific persistence operation, whether it is pusha pattern consumer or pulla pattern consumer, the same logic is executed, that is, it will first judge whether the current consumer is in an available state, that is, RUNNINGthe state, if not, it will not Perform subsequent persistence operations; otherwise, obtain the queue for subsequent persistence operations. pushThe relevant source code of the mode is as follows ( pullthe source code of the mode is similar):
public void persistConsumerOffset() {
try {
// 状态判断
// pull模式的判断是 isRunning()方法
this.makeSureStateOK();
Set<MessageQueue> mqs = new HashSet<MessageQueue>();
Set<MessageQueue> allocateMq = this.rebalanceImpl.getProcessQueueTable().keySet();
mqs.addAll(allocateMq);
this.offsetStore.persistAll(mqs);
} catch (Exception e) {
log.error("group: " + this.defaultMQPushConsumer.getConsumerGroup() + " persistConsumerOffset exception", e);
}
}
// 具体的判断逻辑
private void makeSureStateOK() throws MQClientException {
if (this.serviceState != ServiceState.RUNNING) {
throw new MQClientException("The consumer service state not OK, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
}
}
The subsequent persistence operation is the execution persistAllmethod. There are two situations in the execution of this method, one is local persistent storage, and the other is remote persistent storage.
/**
* Offset store interface
*/
public interface OffsetStore {
/*其他方法*/
/**
* Persist all offsets,may be in local storage or remote name server
*/
void persistAll(final Set<MessageQueue> mqs);
/*其他方法*/
}
In the selection of the persistence mode, BROADCASTINGit will be set to the local persistence mode, and CLUSTERINGwill be set to the remote persistence mode. We will look at the specific two persistence modes later.
local persistence
The local mode persistence will first wrap the offset serialization according to the consumption offset of the current queue, and then directly write the file, =. The source code is as follows:
public void persistAll(Set<MessageQueue> mqs) {
if (null == mqs || mqs.isEmpty())
return;
OffsetSerializeWrapper offsetSerializeWrapper = new OffsetSerializeWrapper();
for (Map.Entry<MessageQueue, AtomicLong> entry : this.offsetTable.entrySet()) {
if (mqs.contains(entry.getKey())) {
AtomicLong offset = entry.getValue();
offsetSerializeWrapper.getOffsetTable().put(entry.getKey(), offset);
}
}
// 序列号json
String jsonString = offsetSerializeWrapper.toJson(true);
if (jsonString != null) {
try {
// 写文件
MixAll.string2File(jsonString, this.storePath);
} catch (IOException e) {
log.error("persistAll consumer offset Exception, " + this.storePath, e);
}
}
}
remote persistence
Remote persistence is to persist the consumption offset of the queue to the corresponding one broker, but there is one more operation, which is to remove the unused queue in the temporary consumption offset table. The source code is as follows:
public void persistAll(Set<MessageQueue> mqs) {
if (null == mqs || mqs.isEmpty())
return;
final HashSet<MessageQueue> unusedMQ = new HashSet<MessageQueue>();
if (!mqs.isEmpty()) {
for (Map.Entry<MessageQueue, AtomicLong> entry : this.offsetTable.entrySet()) {
MessageQueue mq = entry.getKey();
AtomicLong offset = entry.getValue();
if (offset != null) {
if (mqs.contains(mq)) {
try {
this.updateConsumeOffsetToBroker(mq, offset.get());
log.info("[persistAll] Group: {} ClientId: {} updateConsumeOffsetToBroker {} {}",
this.groupName,
this.mQClientFactory.getClientId(),
mq,
offset.get());
} catch (Exception e) {
log.error("updateConsumeOffsetToBroker exception, " + mq.toString(), e);
}
} else {
// 记录没有使用的队列
unusedMQ.add(mq);
}
}
}
}
if (!unusedMQ.isEmpty()) {
for (MessageQueue mq : unusedMQ) {
// 在临时偏移量表中移除未使用的队列
this.offsetTable.remove(mq);
log.info("remove unused mq, {}, {}", mq, this.groupName);
}
}
}
brokerThe direct sending mode is adopted when sending persistent information to the server. If the master node goes down, the information of the slave node will be updated.
/**
* Update the Consumer Offset in one way, once the Master is off, updated to Slave,
* here need to be optimized.
*/
private void updateConsumeOffsetToBroker(MessageQueue mq, long offset) throws RemotingException,
MQBrokerException, InterruptedException, MQClientException {
updateConsumeOffsetToBroker(mq, offset, true);
}
The code that is specifically updated to brokerthis process is a piece of code to be optimized, so there is no need to delve into it, because we already know what it is going to do.