RabbitMQ study notes (Part 2): Delay queue, advanced release confirmation, backup switch

10. Delay queue

delay queue

concept:

Delay queue usage scenarios:

 

flow chart:

Delay queue integration with Springboot

Import dependencies:

<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-amqp</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>com.alibaba</groupId>
			<artifactId>fastjson</artifactId>
			<version>1.2.47</version>
		</dependency>
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
		</dependency>
		<dependency>
			<groupId>io.springfox</groupId>
			<artifactId>springfox-swagger2</artifactId>
			<version>2.9.2</version>
		</dependency>
		<dependency>
			<groupId>io.springfox</groupId>
			<artifactId>springfox-swagger-ui</artifactId>
			<version>2.9.2</version>
		</dependency>

		<dependency>
			<groupId>org.springframework.amqp</groupId>
			<artifactId>spring-rabbit-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

Create the config class SwaggerConfig under java/com/atguigu/rabbitmq and write the code:

@Configuration
@EnableSwagger2
public class SwaggerConfig {
    @Bean
    public Docket webApiConfig(){
        return new Docket(DocumentationType.SWAGGER_2)
           .groupName("webApi")
           .apiInfo(webApiInfo())
           .select()
           .build();
    }
    private ApiInfo webApiInfo() {
        return new ApiInfoBuilder()
           .title("rabbitmq接口文档")
           .description("本文档描述了rabbitmq微服务接口定义")
           .version("1.0")
           .contact(new Contact("enjoy6288","http://atguigu.com","[email protected]"))
           .build();
    }
}

Queue TTL code framework diagram:

Queue TTL (configuration class code):

@Configuration
public class TtlQueueConfig {
    //普通交换机的名称
    public static final String X_EXCHANGE = "X";
    //死信交换机的名称
    public static final String Y_DEAD_LETTER_EXCHANGE = "Y";
    //普通队列的名称
    public static final String QUEUE_A = "QA";
    public static final String QUEUE_B = "QB";
    //死信队列的名称
    public static final String DEAD_LETTER_QUEUE = "QD";
    //声明xExchange
    @Bean("xExchange")
    public DirectExchange xExchange(){
        return new DirectExchange(X_EXCHANGE);
    }
    //声明yExchange
    @Bean("yExchange")
    public DirectExchange yExchange(){
        return new DirectExchange(Y_DEAD_LETTER_EXCHANGE);
    }
    //声明普通队列A的TTL为10s
    @Bean("queueA")
    public Queue queueA(){
        Map<String,Object> arguments = new HashMap<>(3);
        //设置死信交换机
        arguments.put("x-dead-letter-exchange",Y_DEAD_LETTER_EXCHANGE);
        //设置死信RoutingKey
        arguments.put("x-dead-letter-routing-key","YD");
        //设置TTL
        arguments.put("x-message-ttl",10000);
        return QueueBuilder.durable(QUEUE_A).withArguments(arguments).build();
    }
    //声明普通队列B的TTL为40s
    @Bean("queueB")
    public Queue queueB(){
        Map<String,Object> arguments = new HashMap<>(3);
        //设置死信交换机
        arguments.put("x-dead-letter-exchange",Y_DEAD_LETTER_EXCHANGE);
        //设置死信RoutingKey
        arguments.put("x-dead-letter-routing-key","YD");
        //设置TTL
        arguments.put("x-message-ttl",40000);
        return QueueBuilder.durable(QUEUE_B).withArguments(arguments).build();
    }
    //死信队列
    @Bean("queueD")
    public Queue queueD() {
        return QueueBuilder.durable(DEAD_LETTER_QUEUE).build();
    }
    //绑定A-X
    @Bean
    public Binding queueABindingX(@Qualifier("queueA") Queue queueA,
                                 @Qualifier("xExchange") DirectExchange xExchange){
        return BindingBuilder.bind(queueA).to(xExchange).with("XA");
    }
    //绑定B-x
    @Bean
    public Binding queueBBindingX(@Qualifier("queueB") Queue queueB,
                                 @Qualifier("xExchange") DirectExchange xExchange){
        return BindingBuilder.bind(queueB).to(xExchange).with("XB");
    }
    //绑定D-y
    @Bean
    public Binding queueDBindingY(@Qualifier("queueD") Queue queueD,
                                  @Qualifier("yExchange") DirectExchange yExchange){
        return BindingBuilder.bind(queueD).to(yExchange).with("YD");
    }
}

Queue TTL (producer):

@Slf4j
@RestController
@RequestMapping("/ttl")
public class SendMsgController {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    @GetMapping("/sendMsg/{message}")
    public void sendMsg(@PathVariable String message){
        log.info("当前时间：{},发送一条信息给两个TTL队列:{}",new Date().toString(),message);
        rabbitTemplate.convertAndSend("X","XA","消息来自ttl为10s的队列:"+message);
        rabbitTemplate.convertAndSend("X","XB","消息来自ttl为40s的队列:"+message);
    }
}

Queue TTL (consumer)

@Slf4j
@Component
public class DeadLetterQueueConsumer {
    //接收消息
    @RabbitListener(queues="QD")
    public void receiveD(Message message, Channel channel) throws Exception{
        String msg = new String(message.getBody());
        log.info("当前时间：{}，收到死信队列的消息：{}",new Date().toString(),msg);
    }
}

Delay queue optimization

Queues cannot be added for demand,

Write a general queue as a delay queue:

Configuration class, add the following code to the above configuration class:

public static final String QUEUE_C = "QC";
//和死信交换机连接
@Bean("queueC")
public Queue queue(){
    Map<String,Object> arguments = new HashMap<>(3);
    //设置死信交换机
    arguments.put("x-dead-letter-exchange",Y_DEAD_LETTER_EXCHANGE);
    //设置死信RoutingKey
    arguments.put("x-dead-letter-routing-key","YD");
    return QueueBuilder.durable(QUEUE_C).withArguments(arguments).build();
}
//和普通交换机绑定
@Bean
public Binding queueCBindingX(@Qualifier("queueC") Queue queueC,
                              @Qualifier("xExchange") DirectExchange xExchange){
    return BindingBuilder.bind(queueC).to(xExchange).with("XC");
}

Write the following code based on the previous producer: 

@Slf4j
@RestController
@RequestMapping("/ttl")
public class SendMsgController {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    @GetMapping("/sendExpirationMsg/{message}/{ttlTime}")
    public void sendMsg(@PathVariable String message,@PathVariable String ttlTime){
        log.info("当前时间:{},发送一条时长{}毫秒TTL信息给队列QC:{}",
                new Date().toString(),ttlTime,message);
        rabbitTemplate.convertAndSend("X","XC",message,msg->{
            //发送消息的时候延迟时长
            msg.getMessageProperties().setExpiration(ttlTime);
            return msg;
        });
    }
}

Click the startup class to restart, and enter on the web page: localhost:8080/ttl/sendExpirationMsg/Hello1/20000 and localhost:8080/ttl/sendExpirationMsg/Hello2/2000.

Problem: The delay queue is queued. When there are multiple messages in the queue, the messages in the delay queue will be sent based on the longest time in the queue.

Install the delay queue plugin:

Download the rabbitmq_delayed_message_exchange plug-in at https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases, unzip and place it in the RabbitMQ plug-in directory.

The rabbitmq plug-in is in the /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.8/plugins directory. Upload the plug-in to the directory. If the upload fails, use sudo rz to obtain permissions first.

输入：rabbitmq-plugins enable rabbitmq_delayed_message_exchange

Restart rabbitmq: systemctl restart rabbitmq-server

Enter the switch interface of rabbitmq and check whether the following appears. If it appears, it means the installation is successful:

Plugin-based delay queue:

Configuration class

@Configuration
public class DelayedQueueConfig {
    //队列
    public static final String DELAYED_QUEUEE_NAME="delayed.queue";
    //交换机
    public static final String DELAYED_EXCHANGE_NAME="delayed.exchange";
    //routingKey
    public static final String DELAYED_ROUTING_KEY="delayed.routingkey";
    //声明队列
    @Bean
    public Queue delayedQueue(){
        return new Queue(DELAYED_QUEUEE_NAME);
    }
    //声明交换机,基于插件的
    @Bean
    public CustomExchange delayedExchange(){
        Map<String,Object> arguments = new HashMap<>();
        arguments.put("x-delayed-type","direct");
        //交换机名称，交换机类型,是否需要持久化，是否需要自动删除，其它参数
        return new CustomExchange(DELAYED_EXCHANGE_NAME,"x-delayed-message",true,false,arguments);
    }
    //绑定
    @Bean
    public Binding delayedQueueBindingDelayedExchange(@Qualifier("delayedQueue") Queue delayedQueue,
                                                      @Qualifier("delayedExchange") CustomExchange delayedExchange){
        return BindingBuilder.bind(delayedQueue).to(delayedExchange).with(DELAYED_ROUTING_KEY).noargs();
    }
}

producer

//发消息，基于插件的消息及延迟时间
@GetMapping("/sendDelayMsg/{message}/{delayTime}")
public void sendMsg(@PathVariable String message,@PathVariable  Integer delayTime){
    log.info("当前时间:{},发送一条时长{}毫秒信息给延迟队列delayed.queue:{}",
            new Date().toString(),delayTime,message);
    rabbitTemplate.convertAndSend(DelayedQueueConfig.DELAYED_EXCHANGE_NAME,
            DelayedQueueConfig.DELAYED_ROUTING_KEY,message,msg->{
            //发送消息时延迟时间（毫秒）
                msg.getMessageProperties().setDelay(delayTime);
                return msg;
            });
}

Consumer, create DelayQueueConsumer class:

@Slf4j
@Component
public class DelayQueueConsumer {
    //监听消息
    @RabbitListener(queues = DelayedQueueConfig.DELAYED_QUEUEE_NAME)
    public void receiveDelayQueue(Message message){
        String msg = new String(message.getBody());
        log.info("当前时间：{},收到延迟队列的消息:{}",new Date().toString(),msg);
    }

}

Test: localhost:8080/ttl/sendDelayMsg/come on baby1/20000. localhost:8080/ttl/sendDelayMsg/come on baby2/2000

11. Release confirmation advanced

If either the switch or the queue fails, the messages will be lost.

Configuration class:

@Configuration
public class ConfirmConfig {
    //交换机
    public static final String CONFIRM_EXCHANGE_NAME="confirm.exchange";
    //队列
    public static final String CONFIRM_QUEUE_NAME="confirm.queue";
    //RoutingKey
    public static final String CONFIRM_ROUTING_KEY="key1";
    //声明交换机
    @Bean("confirmExchange")
    public DirectExchange confirmExchange(){
        return new DirectExchange(CONFIRM_EXCHANGE_NAME);
    }
    //声明队列
    @Bean("confirmQueue")
    public Queue confirmQueue(){
        return QueueBuilder.durable(CONFIRM_QUEUE_NAME).build();
    }
    //绑定
    @Bean
    public Binding queueBindingExchange(@Qualifier("confirmQueue") Queue confirmQueue,
                                        @Qualifier("confirmExchange") DirectExchange confirmExchange){
        return BindingBuilder.bind(confirmQueue).to(confirmExchange).with(CONFIRM_ROUTING_KEY);
    }
}

Producer:

@RestController
@Slf4j
@RequestMapping("/confirm")
public class ProducerController {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    //发消息
    @GetMapping("/sendMessage/{message}")
    public void sendMessage(@PathVariable String message){
        rabbitTemplate.convertAndSend(ConfirmConfig.CONFIRM_EXCHANGE_NAME,
                ConfirmConfig.CONFIRM_ROUTING_KEY,message);
        log.info("发送消息内容：{}",message);
    }
}

Consumer: 

@Slf4j
@Component
public class Consumer {
    @RabbitListener(queues = ConfirmConfig.CONFIRM_QUEUE_NAME)
    public void receiveConfirmMessagee(Message message){
        String msg = new String(message.getBody());
        log.info("接受到的队列confirm.queue消息：{}",msg);
    }
}

Experimental steps: First enter localhost:8080/confirm/sendMessage/Hello everyone 1.

MyCallBack is an implementation class that inherits the RabbitTemplate.ConfirmCallback interface. Since it is no longer in it, the implementation class cannot be transferred to it when the interface is removed, so it needs to be injected.

Callback interface:

First write the following in the configuration file:

spring.rabbitmq.publisher-confirm-type=correlated

@Slf4j
@Component
public class MyCallBack implements RabbitTemplate.ConfirmCallback {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    @PostConstruct
    public void init(){
        //注入
        rabbitTemplate.setConfirmCallback(this);
    }
    /*交换机确认回调方法
     * 1.发消息，交换机接收到了，回调
     *  1.1.correlationData保存回调消息的ID及相关信息
     *  1.2.交换机收到消息 ack = true
     *  1.3.cause null
     * 2.发消息，交换机接收失败了，回调
     *  2.1.correlationData保存回调信息的ID及相关信息
     *  2.2.交换机收到消息 ack = false
     *  2.3.cause 失败的原因
     */
    @Override
    public void confirm(CorrelationData correlationData,boolean ack,String cause){
        String id = correlationData!=null?correlationData.getId():"";
        if(ack){
            log.info("交换机已经收到了Id为：{}的消息",id);
        }else{
            log.info("交换机已经收到了Id为：{}的消息,由于原因：{}",id,cause);
        }
    }
}

If the switch does not confirm or the confirmation fails, it proves to be a failure and will confirm the callback.

switch confirmation

@RestController
@Slf4j
@RequestMapping("/confirm")
public class ProducerController {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    //发消息
    @GetMapping("/sendMessage/{message}")
    public void sendMessage(@PathVariable String message){
        CorrelationData correlationData = new CorrelationData("1");
        rabbitTemplate.convertAndSend(ConfirmConfig.CONFIRM_EXCHANGE_NAME,
                ConfirmConfig.CONFIRM_ROUTING_KEY,message,correlationData);
        log.info("发送消息内容：{}",message);
    }
}

Enter: localhost:8080/confirm/sendMessage/Hello everyone 1, test callback.

Fallback message

The situation of returning is when an error occurs in the routing key, which is basically a problem with the queue, and the error message needs to be rolled back.

Mandatory parameters:

First add the following code to the configuration file:

spring.rabbitmq.publisher-returns=true

 The code is modified based on MyCallBack above. First add RabbitTemplate.ReturnCallback after implements. Remember to inject setReturnCallback in init. ReturnedMessage is the overriding method. Click MyCallBack and hold down alt+enter to rewrite.

@Slf4j
@Component
public class MyCallBack implements RabbitTemplate.ConfirmCallback,RabbitTemplate.ReturnCallback {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    @PostConstruct
    public void init(){
        //注入
        rabbitTemplate.setConfirmCallback(this);
        rabbitTemplate.setReturnCallback(this);
    }
    //可以在当消息传递过程中不可达目的地时将消息返回给生产者
    //只有不可达目的地的时候，才进行回退
    @Override
    public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
        log.error("消息{},被交换机{}退回，退回原因：{}，路由Key：{}",
                new String(message.getBody()),exchange,replyText,routingKey);
    }
}

12. Backup switch

You need to write a backup switch, a backup queue, and an alarm queue.

Configure the ConfirmConfig class, and be sure to [modify the confirmation switch] to associate it with the backup switch; be sure to add @Bean to the binding method:

//声明确认交换机
@Bean("confirmExchange")
public DirectExchange confirmExchange(){
    return ExchangeBuilder.directExchange(CONFIRM_EXCHANGE_NAME).durable(true)
            .withArgument("alternate-exchange",BACKUP_EXCHANGE_NAME).build();
}
public static final String BACKUP_EXCHANGE_NAME = "backup_exchange";
public static final String BACKUP_QUEUE_NAME="backup_queue";
public static final String WARNING_QUEUE_NAME="warning_queue";
//备份交换机
@Bean("backupExchange")
public FanoutExchange backupExchange(){
    return new FanoutExchange(BACKUP_EXCHANGE_NAME);
}
//备份队列
@Bean("backupQueue")
public Queue backupQueue(){
    return QueueBuilder.durable(BACKUP_QUEUE_NAME).build();
}
//报警队列
@Bean("warningQueue")
public Queue warningQueue(){
    return QueueBuilder.durable(WARNING_QUEUE_NAME).build();
}
//绑定
@Bean
public Binding backupQueueBindingBackupExchange(@Qualifier("backupQueue") Queue backupQueue,
                                    @Qualifier("backupExchange") FanoutExchange backupExchange){
    return BindingBuilder.bind(backupQueue).to(backupExchange);
}
@Bean
public Binding warningQueueBindingBackupExchange(@Qualifier("warningQueue") Queue warningQueue,
                                                @Qualifier("backupExchange") FanoutExchange backupExchange){
    return BindingBuilder.bind(warningQueue).to(backupExchange);
}

Result analysis:

Create a WarningConsumer class under consumer and write the following code:

@Component
@Slf4j
public class WarningConsumer {
    //接收报警消息
    @RabbitListener(queues = ConfirmConfig.WARNING_QUEUE_NAME)
    public void receiveWarningMsg(Message message){
        String msg = new String(message.getBody());
        log.error("报警发现不可路由消息：{}",msg);
    }
}

Then delete the [confirm switch] in the rabbitmq visual interface:

If you want to achieve the following effect, you need to change the value of routingkey, such as adding a "2".

It can be seen that the priority of the backup switch is higher than the rollback message.

Idempotence:

concept:

Repeated consumption of messages:

Solutions:

Idempotence guarantee on the consumer side:

1. Unique ID + fingerprint code mechanism

2. Take advantage of Redis atomicity

Priority queue usage scenarios:

The larger the priority queue is from 0 to 255, the higher the priority.

Priority queue code implementation:

The producer code is as follows:

public class producer {
    //队列名称
    public static final String QUEUE_NAME = "hello";
    //发消息
    public static void main( String[] args ) throws IOException, TimeoutException {
        //第1步：创建一个连接工程
        ConnectionFactory factory = new ConnectionFactory();
        //第2步：输入工厂IP，用户名和密码——连接RabbitMQd队列
        factory.setHost("192.168.182.157");
        factory.setUsername("admin");
        factory.setPassword("123");
        //第3步：创建连接
        Connection connection = factory.newConnection();
        //第4步：获取信道
        Channel channel = connection.createChannel();
        Map<String,Object> arguments = new HashMap<>();
        arguments.put("x-max-priority",10);
        channel.queueDeclare(QUEUE_NAME,true,false,false,arguments);
        for(int i=1;i<11;i++){
            String message = "info"+i;
            if(i==5){
                AMQP.BasicProperties properties = new AMQP.BasicProperties().builder().priority(5).build();
                channel.basicPublish("",QUEUE_NAME,properties,message.getBytes());channel.basicPublish("",QUEUE_NAME,null,message.getBytes());
            }else{
                channel.basicPublish("",QUEUE_NAME,null,message.getBytes());
            }
        }
    }
}

Test: The producer first sends all messages to the queue, and then the consumers consume them. Start the producer first, then the consumer. Finally, info5 is at the top of the console and is printed last, which is logical.

13. Cluster

Lazy queue:

Whether messages are kept in memory or on disk. Normal case: Messages are kept in memory. Lazy queue: messages are saved on disk. Lazy queue consumption is very slow and needs to be fetched into memory first. Usage scenario: There are too many messages accumulated, such as 1 million, but the consumer is down.

scenes to be used:

Two modes:

Cluster principle:

Build a cluster:

Shut down first, clone 2 machines, and then start them all. Check the ip address, and then use xshell to connect to 3 hosts.

 

Step 1: Modify the host names of 3 machines. Enter hostname on the first host and node1 is displayed. Enter vi /etc/hostname on the second and third hosts, and then change node1 to node2 and node3 respectively. Enter [reboot] to restart.

Step 2: Configure the hosts file of each node so that each node can recognize each other. vi /etc/hosts. Assign the three ip node values ​​to three Xshell sessions.

Step 3: Make sure that the cookie files of each node use the same value. Execute remote operation command on node1:

scp /var/lib/rabbitmq/.erlang.cookie root@node2:/var/lib/rabbitmq/.erlang.cookie
scp /var/lib/rabbitmq/.erlang.cookie root@node3:/var/lib/rabbitmq/.erlang.cookie

Step 4: Start the RabbitMQ service, and also start the Erlang virtual machine and RabbitMQ application service (execute the following commands on the three nodes): rabbitmq-server -detached

Step 5: Execute on node 2:

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app

Step 6: Execute on node 3:

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@node2
rabbitmqctl start_app

Step 7: Check the cluster status: rabbitmqctl cluster_status

Step 8: Reset the user: Create an account: rabbitmqctl add_user admin 123. Set user role: rabbitmqctl set_user_tags admin administrator. Set user permissions: rabbitmqctl set_permissions -p "/" admin ".*" ".*" ".*"

Step 9: Deactivate the cluster nodes (executed separately on node2 and node3 machines) 

Mirror queue:

Mirroring is a backup, and the received message should not only exist on one node, otherwise if one node goes down, the message will disappear.

ha-mode: standby mode. ha-param: Fill in the backup quantity. ha-sync-mode backup mode.

Experiment: Create a queue on machine No. 1, and then automatically back up a copy on machine No. 2 or No. 3. Now node 1 is shut down and the backup data will be stored in another

To achieve high availability load balancing:

VIP connects to the host. If the host goes down, the connection will automatically drift to the backup machine. The standby machine will also ask for the existence of the host machine from time to time. Requires Haproxy and keepalive software.

If the high-availability host fails, the backup machine will take over.

14. Federation

1. First, open multiple nodes, and then let each node run independently.

2. Enable the federation plug-in on each machine: first enter: rabbitmq-plugins enable rabbitmq_federation and then enter: rabbitmq-plugins enable rabbitmq_federation_management

Exchange implementation:

Here is part of the code:

public static final String FED_EXCHANGE="fed_exchange";
channel.exchangeDeclare(FED_EXCHANGE,BuiltinExchangeType.DIRECT);
channel.queueDeclare("node2_queue",true,false,false,null);
channel.queueBind("node2_queue",FED_EXCHANGE,"routeKey");

The following is the specific configuration:

 

Check whether it is successful. If successful, the following appears:

Queue implementation:

Shovel：