四、RabbitMQ 延时消费、 批量消费和消息重试

本文重点介绍三方面内容

• 消息延迟
  消息延迟通俗来说就是用队列来存放需要在指定时间被处理的消息元素。消息延迟的使用场景也比较多，比如订单在十分钟之内未支付就自动取消。

• 消息重试
  

消息重试的意义在于尽可能降低非业务异常情况下错误数据的产生。比如系统重启、依赖的第三方服务有问题等情况时，可以利用重试来确保自己的业务不受影响。

• 批量消费
  批量消费主要的目的就是提高消费速率，降低频繁的去Broker上拉取消息，可一次拉去多条，然后针对该批次消息进行处理。

1. 延迟消费

  要实现消息的延迟消费，首先要理解 RabbitMQ 中的 TTL ，TTL 是 RabbitMQ 中一个消息或者队列的属性，表明一条消息或者该队列中的所有消息的最大存活时间，存活时间单位是毫秒。

• queue设置 TTL

@Bean(name = "ackQueue")
public Queue ackQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {
    Queue queue = new Queue(RabbitMqConfig.queue, true);
    queue.addArgument("x-message-ttl",1000);
    rabbitAdmin.declareQueue(queue);
    return queue;
}

• 每条消息设置 TTL

rabbitTemplate.convertAndSend("exchange", "routingKey", "mesage body", messagePostProcessor -> {
    messagePostProcessor.getMessageProperties().setExpiration("1000");
     return messagePostProcessor;
});

注意：

• 队列设置 TTL 属性后，一旦消息过期，就会被队列丢弃(如果配置了死信队列被丢到死信队列中)。
• 消息设置 TTL 属性后，不一定会被马上丢弃，因为消息是否过期是在即将投递到消费者之前判定的，如果当前队列消息积压严重，那么已过期的消息还能存活较长时间。另外，如果不设置 TTL，表示消息永远不会过期。

死信 + TTL

RabbitMQ 延时消费必须依赖 死信队列 和 TTL 才能实现。可理解为 TTL 能让消息在一定时间后成为死信（ 三、RabbitMQ 的死信队列），成为死信的消息会被重新投递到死信队列里面，这样消费者一直消费死信队列的消息就完成了延时消费的目的。

废话不多说，根据这个结构图开始撸代码！

(1) RabbitMQ基础配置

@Configuration
public class RabbitmqBaseConfig {

    @Value("${spring.rabbitmq.addresses}")
    private String address;
    @Value("${spring.rabbitmq.username}")
    private String username;
    @Value("${spring.rabbitmq.password}")
    private String password;

    // 生产者发送消息连接
    @Bean("connectionFactory")
    public ConnectionFactory connectionFactory() {
        CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
        connectionFactory.setAddresses(address);
        connectionFactory.setShuffleAddresses(true);
        connectionFactory.setUsername(username);
        connectionFactory.setPassword(password);
        return connectionFactory;
    }

    @Bean(name = "rabbitAdmin")
    public RabbitAdmin rabbitAdmin() {
        RabbitAdmin rabbitAdmin = new RabbitAdmin(connectionFactory());
        rabbitAdmin.setAutoStartup(true);
        return rabbitAdmin;
    }
    
    // 消费者连接工厂配置
    @Bean("rabbitListenerFactory")
    public RabbitListenerContainerFactory<?> rabbitListenerContainerFactory(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
        SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
        factory.setMessageConverter(new Jackson2JsonMessageConverter());
        // 消费者自动确认消息
        factory.setAcknowledgeMode(AcknowledgeMode.AUTO);
        factory.setConnectionFactory(connectionFactory);
        factory.setPrefetchCount(1);
        return factory;
    }

}

(2) 绑定关系配置

@Configuration
public class RabbitmqDelayConfig {

    /*业务队列*/
    public static final String normalExchange = "exchange.cal.center";
    public static final String updateRoutingKey = "rk.update";
    public static final String updateQueue = "queue.update.info";

    /*死信队列*/
    public static final String deadLetterExchange = "exchange.dead.letter";
    public static final String deadUpdateRoutingKey = "rk.dead.update";
    public static final String delayQueue = "queue.delay.update";
  

    // 生产者发送消息配置
    @Bean(name = "rabbitTemplate")
    public AmqpTemplate rabbitTemplate(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
        RabbitTemplate template = new RabbitTemplate(connectionFactory);
        template.setExchange(normalExchange);
        template.setUsePublisherConnection(true);
        template.setMessageConverter(new Jackson2JsonMessageConverter());
        return template;
    }

    // 声明业务交换机
    @Bean(name = "normalExchange")
    TopicExchange normalExchange() {
        return ExchangeBuilder.topicExchange(normalExchange).durable(true).build();
    }

    // 声明业务队列，并给业务队列 updateQueue 指定死信交换机 和 routingKey
    @Bean("updateQueue")
    public Queue updateQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {

        Queue queue = new Queue(updateQueue, true);
        queue.addArgument("x-dead-letter-exchange", deadLetterExchange);
        queue.addArgument("x-dead-letter-routing-key", deadUpdateRoutingKey);
        queue.addArgument("x-message-ttl", 20000); // 延迟 2s

        rabbitAdmin.declareQueue(queue);
        return queue;
    }


    // 声明业务队列 和 交换机绑定的 routingKey
    @Bean
    public Binding employeeUpdateQueueBinding(@Qualifier("updateQueue") Queue updateQueue, @Qualifier("normalExchange") TopicExchange normalExchange) {
        return BindingBuilder.bind(updateQueue).to(normalExchange).with(updateRoutingKey);
    }

    /*******************************************************************************************************/

    // 声明死信交换机
    @Bean("deadLetterExchange")
    public TopicExchange deadExchange() {
        return ExchangeBuilder.topicExchange(deadLetterExchange).durable(true).build();
    }

    // 声明死信队列
    @Bean(name = "delayQueue")
    public Queue delayQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {
        Queue queue = new Queue(delayQueue, true);
        rabbitAdmin.declareQueue(queue);
        return queue;
    }

    // 声明死信队列 和 交换机绑定的 routingKey
    @Bean
    public Binding deadBinding(@Qualifier("delayQueue") Queue delayQueue, @Qualifier("deadLetterExchange") TopicExchange deadLetterExchange) {
        return BindingBuilder.bind(delayQueue).to(deadLetterExchange).with(deadUpdateRoutingKey);
    }
}

(3) 消息生产者

@Slf4j
@RestController
@RequestMapping("/delay")
public class Producer {
    @Resource
    private RabbitTemplate rabbitTemplate;

    @PostMapping("/send")
    public void send(int i) {

        String message = "delay msg " + i;
        rabbitTemplate.convertAndSend(RabbitmqDelayConfig.normalExchange, RabbitmqDelayConfig.updateRoutingKey, message);
        log.info("消息发送完成 : " + message);
    }
}

(4) 消费者

@Slf4j
@Component
public class DelayUpdateConsumer {

    @RabbitHandler
    @RabbitListener(queues = RabbitmqDelayConfig.delayQueue, containerFactory = "rabbitListenerFactory")
    public void receiveMessage(Message message) {
        String msgBody = new String(message.getBody(), StandardCharsets.UTF_8);
        log.info("接收到 " + RabbitmqDelayConfig.delayQueue + " 队列消息 : {}", msgBody);
    }
}

(5) 启动类

@EnableRabbit
@SpringBootApplication(scanBasePackages = {"com.sff.delay.batch.example"}, exclude = {RabbitAutoConfiguration.class})
public class RabbitmqDelayBatchApplication {

    public static void main(String[] args) {
        SpringApplication.run(RabbitmqDelayBatchApplication.class, args);
    }
}

(6) 结果验证

  从运行日志上看到死信队列是在 2s 后收到了生产者的消息，从 RabbitMQ后台来看消息达到queue.update.info 后没有立马被消费。

2. 消息重试

2.1 利用RabbitMQ手动确认 和 延迟队列的特性来实现消息重试

注意一点，死信 exchange 就是你的业务 exchange

(1) 消费模式改成手动确认

// 消费者连接工厂配置
@Bean("rabbitListenerFactory")
public RabbitListenerContainerFactory<?> rabbitListenerContainerFactory(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
    SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
    factory.setMessageConverter(new Jackson2JsonMessageConverter());
    // 消费者自动确认消息
    factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
    factory.setConnectionFactory(connectionFactory);
    factory.setPrefetchCount(1);
    return factory;
}

(1) 声明绑定关系

@Configuration
public class RabbitmqRetryConfig {

    /*业务队列*/
    public static final String retryBizExchange = "exchange.retry";
    public static final String retryBizRoutingKey = "rk.retry";
    public static final String retryBizQueue = "queue.retry.info";

    //  延迟交换机,用于消息重试使用
    public static final String delayExchange = "exchange.delay";
    public static final String delayRetryQueue = "queue.delay.retry";


    // 生产者发送消息配置
    @Bean(name = "rabbitTemplate")
    public AmqpTemplate rabbitTemplate(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
        RabbitTemplate template = new RabbitTemplate(connectionFactory);
        template.setExchange(retryBizExchange);
        template.setUsePublisherConnection(true);
        template.setMessageConverter(new Jackson2JsonMessageConverter());
        return template;
    }

    // 声明业务交换机
    @Bean(name = "retryBizExchange")
    TopicExchange retryBizExchange() {
        return ExchangeBuilder.topicExchange(retryBizExchange).durable(true).build();
    }

    // 声明名业队列
    @Bean(name = "retryBizQueue")
    public Queue retryBizQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {
        Queue queue = new Queue(retryBizQueue, true);
        rabbitAdmin.declareQueue(queue);
        return queue;
    }

    // 声明业务队列 和 交换机绑定的 routingKey
    @Bean
    public Binding employeeUpdateQueueBinding(@Qualifier("retryBizQueue") Queue updateQueue, @Qualifier("retryBizExchange") TopicExchange normalExchange) {
        return BindingBuilder.bind(updateQueue).to(normalExchange).with(retryBizRoutingKey);
    }

    // 延迟重试队列
    @Bean(name = "delayRetryQueue")
    public Queue delayRetryQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {
        Queue queue = new Queue(delayRetryQueue, true);
        // 将消息重新投递到业务 exchange
        queue.addArgument("x-dead-letter-exchange", retryBizExchange);

        // 整个消息队列设置延迟指定时候被丢弃成为死信，消息会重新投递到 x-dead-letter-exchange 对应的队列中，routingKey为自己指定
        queue.addArgument("x-message-ttl", 10000);

        rabbitAdmin.declareQueue(queue);
        return queue;
    }

    // 延迟重试交换机
    @Bean(name = "delayRetryExchange")
    TopicExchange delayRetryExchange() {
        return ExchangeBuilder.topicExchange(delayExchange).durable(true).build();
    }

    // 延迟交换机 exchange  和  业务 routingKey 绑定
    @Bean
    public Binding retryBinding(@Qualifier("delayRetryQueue") Queue queue,
                                        @Qualifier("delayRetryExchange") TopicExchange exchange) {
        return BindingBuilder.bind(queue).to(exchange).with(retryBizRoutingKey);
    }
}

(3) 重试策略封装

public abstract class AbstractRetryMessageListener {

    protected String delayRetryExchange = "";
    protected String bizRoutingKey = "";
    protected Class loggerClass;

    public void onMessage(Message message, Channel channel) throws Exception {

        int maxRetryTimes = getMaxRetryNum();
        String messageId = message.getMessageProperties().getMessageId();

        boolean processResult = false;
        try {

            processResult = onMsg(message);

        } catch (Exception e) {

            String errorMsg = e.getMessage() == null ? "" : e.getMessage();
            getLogger().warn("MQ消费异常 , messageId: {}, errorMsg: {}", messageId, errorMsg, e);

        } finally {

            if (processResult) {
                // 消息消费成功，手动 ack
                this.basicAck(channel, message);

            } else {

                int retryNum = this.getRetryNum(message);
                if (retryNum <= maxRetryTimes) {
                    // 消息重新投递到队列中进行二次消费
                    this.basicPublish(channel, message);
                    this.basicAck(channel, message);

                } else {

                    getLogger().error("当前消息体重试操作超过 {} 次, 放弃重试! messageId: {}, retryNum: {}", maxRetryTimes, messageId, retryNum);
                    this.basicAck(channel, message);
                }
            }
        }
    }

    private void basicAck(Channel channel, Message message) throws Exception {
        channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
        return;
    }

    /**
     * 消息重新投递到队列中去
     */
    private void basicPublish(Channel channel, Message message) throws Exception {
        MessagePropertiesConverter messagePropertiesConverter = new DefaultMessagePropertiesConverter();
        AMQP.BasicProperties basicProperties = messagePropertiesConverter.fromMessageProperties(message.getMessageProperties(), "utf-8");
        channel.basicPublish(delayRetryExchange, bizRoutingKey, basicProperties, message.getBody());
    }

    private int getRetryNum(Message message) {
        int retryNum = 1;

        try {
            Map<String, Object> headers = message.getMessageProperties().getHeaders();
            if (headers == null || !headers.containsKey("x-death")) {
                return retryNum;
            }

            List<Map<String, Object>> deathList = (List<Map<String, Object>>) headers.get("x-death");
            if (deathList == null || deathList.size() == 0) {
                return retryNum;
            }

            Map<String, Object> deathMap = deathList.get(0);
            // 获取重试次数
            Long deathCount = (Long) deathMap.get("count");
            if (deathCount == null || deathCount == 0) {
                return retryNum;
            }

            return deathCount.intValue();

        } catch (Exception e) {

            String messageId = message.getMessageProperties().getMessageId();
            getLogger().error("获取消息重试次数异常, messageId: {}", messageId, e);
        }
        return retryNum;
    }

    private Logger getLogger() {
        if (loggerClass == null) {
            return LoggerFactory.getLogger(this.getClass());
        } else {
            return LoggerFactory.getLogger(loggerClass);
        }
    }

    /**
     * 消息业务处理
     * @param message 消息体
     * @throws Exception
     */
    protected abstract boolean onMsg(Message message);

    /**
     * 消息重试次数
     * @return
     */
    protected abstract int getMaxRetryNum();
}

(4) 消息生产者

@Slf4j
@RestController
@RequestMapping("/retry")
public class RetryProducer {
    @Resource
    private RabbitTemplate rabbitTemplate;

    @PostMapping("/send")
    public void send(int i) {

        Map<String, String> msgObject = new HashMap<>();
        msgObject.put("content", "retry message " + i);
        String msgBody = JacksonUtil.toJson(msgObject);

        String messageId = new AlternativeJdkIdGenerator().generateId().toString().replace("-", "");
        Message message = MessageBuilder
                .withBody(msgBody.getBytes())
                .setMessageId(messageId)
                .setTimestamp(new Date()).build();

        rabbitTemplate.convertAndSend(RabbitmqRetryConfig.retryBizExchange, RabbitmqRetryConfig.retryBizRoutingKey, message);
        log.info("消息发送完成 : " + message);
    }
}

(5) 消息消费者

@Slf4j
@Component
public class RetryConsumer extends AbstractRetryMessageListener {

    @RabbitHandler
    @RabbitListener(queues = RabbitmqRetryConfig.retryBizQueue, containerFactory = "rabbitListenerFactory")
    public void receiveMessage(Message message, Channel channel) throws Exception {

        this.loggerClass = RetryConsumer.class;
        this.delayRetryExchange = RabbitmqRetryConfig.delayExchange;
        this.bizRoutingKey = RabbitmqRetryConfig.retryBizRoutingKey;

        super.onMessage(message, channel);
    }

    @Override
    protected boolean onMsg(Message message) {

        String msgBody = new String(message.getBody(), StandardCharsets.UTF_8);
        log.info("接收到 " + RabbitmqRetryConfig.retryBizQueue + " 队列消息 : {}", msgBody);

        // 模拟业务处理异常
        throw new NullPointerException();
    }

    @Override
    protected int getMaxRetryNum() {
        return 3;
    }
}

3. 批量消费

批量消费的目的是一次去Broker拉取多条消息，然后程序一条条处理，并手动确认即可。参考官方文档 @RabbitListener with Batching

重点设置 SimpleRabbitListenerContainerFactory 的三个参数：

factory.setConsumerBatchEnabled(true);
factory.setBatchListener(true);
factory.setBatchSize(3);

3.1 声明绑定关系

@Configuration
public class RabbitmqBatchConfig {

    /*业务队列*/
    public static final String batchBizExchange = "exchange.batch";
    public static final String batchBizRoutingKey = "rk.batch";
    public static final String batchBizQueue = "queue.batch.info";

    // 生产者发送消息配置
    @Bean(name = "rabbitTemplate")
    public AmqpTemplate rabbitTemplate(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
        RabbitTemplate template = new RabbitTemplate(connectionFactory);
        template.setExchange(batchBizExchange);
        template.setUsePublisherConnection(true);
        template.setMessageConverter(new Jackson2JsonMessageConverter());
        return template;
    }

    // 声明业务交换机
    @Bean(name = "batchBizExchange")
    TopicExchange batchBizExchange() {
        return ExchangeBuilder.topicExchange(batchBizExchange).durable(true).build();
    }

    // 声明名业队列
    @Bean(name = "batchBizQueue")
    public Queue batchBizQueue(@Qualifier("rabbitAdmin") RabbitAdmin rabbitAdmin) {
        Queue queue = new Queue(batchBizQueue, true);
        rabbitAdmin.declareQueue(queue);
        return queue;
    }

    // 声明业务队列 和 交换机绑定的 routingKey
    @Bean
    public Binding batchBinding(@Qualifier("batchBizQueue") Queue batchBizQueue, @Qualifier("batchBizExchange") TopicExchange batchBizExchange) {
        return BindingBuilder.bind(batchBizQueue).to(batchBizExchange).with(batchBizRoutingKey);
    }

    // 批量消费连接设置
    @Bean("batchRabbitListenerFactory")
    public RabbitListenerContainerFactory<?> rabbitListenerContainerFactory(@Qualifier("connectionFactory") ConnectionFactory connectionFactory) {
        SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
        factory.setMessageConverter(new Jackson2JsonMessageConverter());
        // 消费者自动确认消息
        factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
        factory.setConnectionFactory(connectionFactory);

        // 启动批量拉取消息
        factory.setConsumerBatchEnabled(true);
        factory.setBatchListener(true);
        factory.setBatchSize(3);
        return factory;
    }
}

3.2 消息消费者

@Slf4j
@Component
public class BatchConsumer {

    @RabbitHandler
    @RabbitListener(queues = RabbitmqBatchConfig.batchBizQueue, containerFactory = "batchRabbitListenerFactory")
    public void receiveMessage(List<Message> messages, Channel channel) throws Exception {

        log.info("接受到该批次消息长度, {} ", messages.size());
        for (Message message : messages) {
            String body = new String(message.getBody(), StandardCharsets.UTF_8);

            log.info("接受到消息, {} ", body);
            // 手动ack
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
        }
    }
}

3.3 消息生产者

@PostMapping("/batch")
public void batch() {

    for (int i = 1; i < 50; i++) {

        Map<String, String> msgObject = new HashMap<>();
        msgObject.put("content", "batch message " + i);
        String msgBody = JacksonUtil.toJson(msgObject);

        String messageId = new AlternativeJdkIdGenerator().generateId().toString().replace("-", "");
        Message message = MessageBuilder
                .withBody(msgBody.getBytes())
                .setMessageId(messageId)
                .setTimestamp(new Date()).build();

        rabbitTemplate.convertAndSend(RabbitmqBatchConfig.batchBizExchange, RabbitmqBatchConfig.batchBizRoutingKey, message);
        log.info("消息发送完成 : " + message);
    }
}

四、写在最后

• 本文案例使用的 spring boot 版本 2.2.10.RELEASE
• 案例代码
• 我正在参与掘金技术社区创作者签约计划招募活动，点击链接报名投稿。