RocketMq简介及重要概念

编程语言 2018-11-04 15:40:34 阅读次数: 0

目录

RocketMq简介

RocketMq重要概念

单机RocketMq搭建

RocketMq总体结构

客户端寻址方式

RocketMq简介

rocketmq是一个消息中间件，基于Kafka的设计思想，但不是kafka的拷贝，它具有高吞吐量，高可用性，适用于大规模分布式系统的特点

RocketMq重要概念

1.producer

消息生产者

2.Consumer

消息消费者

扫描二维码关注公众号，回复： 3927346 查看本文章

3.PushConsumer

broker推送消息到consumer

4.PullConsumer

consumer主动从broker定时pull消息

5.ProducerGroup

一类producer的集合名称，这类producer通常发送一类消息，且发送逻辑一致

6.ConsumerGroup

一类consumer的集合名称，这类consumer通常消费一类消息，且消费逻辑一致

7.broker

消息中转站

8.nameserve

无状态的数据节点，记录broker的路由信息，以及topic，队列等

9.广播消息

一个消息被多个consumer消费，即使这些consumer属于同一个组，也会被组内的每个consumer都消费一次

10.集群消息

一个consumergroup中的consumer实例平均分摊消费消息,即不需要自己做消息消费的负载均衡，只需要扩展机器即可

11.Topic

表示一个类别

12.tag

Topic下的一个子类消息,更进一步细分消息类型

单机RocketMq搭建

1.上传rocketmq jar包

2.tar zxvf 解压rocketmq

3.修改虚拟机参数

/bin/runserver.sh /bin/runbroker.sh 中的虚拟机参数，因为默认4g，可以改小点

JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"

4.创建log日志

在conf下更改日志打印路径

5.启动nameserver和broker

启动nameserver

nohup sh bin/mqnamesrv &

启动broker

nohup sh bin/mqbroker -n localhost:9876 &

查看连接此nameserve的broker

sh mqadmin clusterList -n localhost:9876

结果

可以看到此nameserve只有一个broker注册

测试消息的发送和接受

生产者代码:

public class FirstSyncProducer {
    public static  void main(String[] args) throws Exception{
        //Instantiate with a producer group name.
        Defau ltMQProducer producer = new
                DefaultMQProducer("please_rename_unique_group_name");
        // Specify name server addresses.
        producer.setNamesrvAddr("192.168.122.10:9876");
        producer.setVipChannelEnabled(false);
        //Launch the instance.
        producer.start();
        for (int i = 0; i < 10; i++) {
            //Create a message instance, specifying topic, tag and message body.
            Message msg = new Message("TopicTest" /* Topic */,
                    "TagA" /* Tag */,
                    ("Hello RocketMQ " +
                            i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
            );
            //Call send message to deliver message to one of brokers.
            SendResult sendResult = producer.send(msg);
            System.out.printf("%s%n", sendResult);
        }
        //Shut down once the producer instance is not longer in use.
        producer.shutdown();
    }
}

发送结果:

​
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC7255F0000, offsetMsgId=C0A87A0A00002A9F0000000000000000, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=2], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725AB0001, offsetMsgId=C0A87A0A00002A9F00000000000000B2, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=3], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725B50002, offsetMsgId=C0A87A0A00002A9F0000000000000164, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=0], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725C20003, offsetMsgId=C0A87A0A00002A9F0000000000000216, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=1], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725CE0004, offsetMsgId=C0A87A0A00002A9F00000000000002C8, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=2], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725D60005, offsetMsgId=C0A87A0A00002A9F000000000000037A, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=3], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725DE0006, offsetMsgId=C0A87A0A00002A9F000000000000042C, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=0], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725E70007, offsetMsgId=C0A87A0A00002A9F00000000000004DE, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=1], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725F50008, offsetMsgId=C0A87A0A00002A9F0000000000000590, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=2], queueOffset=2]
SendResult [sendStatus=SEND_OK, msgId=0A011949442418B4AAC27EC725FF0009, offsetMsgId=C0A87A0A00002A9F0000000000000642, messageQueue=MessageQueue [topic=TopicTest, brokerName=bogon, queueId=3], queueOffset=2]

​

消费者代码:

public class FirstConsumer {

    public static void main(String[] args) throws InterruptedException, MQClientException {

        // Instantiate with specified consumer group name.
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name");

        // Specify name server addresses.
        consumer.setNamesrvAddr("192.168.122.10:9876");
        consumer.setVipChannelEnabled(false);
        // Subscribe one more more topics to consume.
        consumer.subscribe("TopicTest", "*");
        // Register callback to execute on arrival of messages fetched from brokers.
        consumer.registerMessageListener(new MessageListenerConcurrently() {

            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                                                            ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });

        //Launch the consumer instance.
        consumer.start();

        System.out.printf("Consumer Started.%n");
    }
}

结果:

ConsumeMessageThread_9 Receive New Messages: [MessageExt [queueId=2, storeSize=178, queueOffset=1, sysFlag=0, bornTimestamp=1540450180558, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180559, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F00000000000002C8, commitLogOffset=712, bodyCRC=601994070, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=3, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725CE0004, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_8 Receive New Messages: [MessageExt [queueId=3, storeSize=178, queueOffset=2, sysFlag=0, bornTimestamp=1540450180607, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180613, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F0000000000000642, commitLogOffset=1602, bodyCRC=1565577195, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=3, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725FF0009, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_5 Receive New Messages: [MessageExt [queueId=3, storeSize=178, queueOffset=1, sysFlag=0, bornTimestamp=1540450180566, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180568, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F000000000000037A, commitLogOffset=890, bodyCRC=1424393152, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=3, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725D60005, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_6 Receive New Messages: [MessageExt [queueId=0, storeSize=178, queueOffset=1, sysFlag=0, bornTimestamp=1540450180574, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180576, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F000000000000042C, commitLogOffset=1068, bodyCRC=1307562618, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=2, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725DE0006, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_4 Receive New Messages: [MessageExt [queueId=1, storeSize=178, queueOffset=1, sysFlag=0, bornTimestamp=1540450180583, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180584, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F00000000000004DE, commitLogOffset=1246, bodyCRC=988340972, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=2, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725E70007, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_1 Receive New Messages: [MessageExt [queueId=1, storeSize=178, queueOffset=0, sysFlag=0, bornTimestamp=1540450180546, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180547, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F0000000000000216, commitLogOffset=534, bodyCRC=1032136437, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=2, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725C20003, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_3 Receive New Messages: [MessageExt [queueId=0, storeSize=178, queueOffset=0, sysFlag=0, bornTimestamp=1540450180533, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180538, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F0000000000000164, commitLogOffset=356, bodyCRC=1250039395, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=2, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725B50002, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_7 Receive New Messages: [MessageExt [queueId=2, storeSize=178, queueOffset=0, sysFlag=0, bornTimestamp=1540450180448, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180486, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F0000000000000000, commitLogOffset=0, bodyCRC=613185359, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=3, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC7255F0000, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_2 Receive New Messages: [MessageExt [queueId=3, storeSize=178, queueOffset=0, sysFlag=0, bornTimestamp=1540450180523, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180522, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F00000000000000B2, commitLogOffset=178, bodyCRC=1401636825, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=3, CONSUME_START_TIME=1540450248729, UNIQ_KEY=0A011949442418B4AAC27EC725AB0001, WAIT=true, TAGS=TagA}, body=16]]] 
ConsumeMessageThread_10 Receive New Messages: [MessageExt [queueId=2, storeSize=178, queueOffset=2, sysFlag=0, bornTimestamp=1540450180597, bornHost=/192.168.122.1:5393, storeTimestamp=1540450180600, storeHost=/192.168.122.10:10911, msgId=C0A87A0A00002A9F0000000000000590, commitLogOffset=1424, bodyCRC=710410109, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message [topic=TopicTest, flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=3, CONSUME_START_TIME=1540450248730, UNIQ_KEY=0A011949442418B4AAC27EC725F50008, WAIT=true, TAGS=TagA}, body=16]]]

从上面看到，消息是消费成功的，并且默认的topic是四个队列

RocketMq总体结构

Broker

a.连接

broker与每个nameserve保持长连接

b.心跳

心跳间隔:broker每隔30秒向所有的nameserve发送心跳,心跳包含topic信息

心跳超时:nameserver间隔10秒钟扫描所有还存活的broker连接，若某个broker2分钟内没有发送数据，则断开连接

c.可用性

rocketmq一般都是使用master/slave结构,slave定期从master读取数据,一旦master挂掉，消费者定期从slave消费数据,但slave不能写入数据

d.可靠性

发送到broker的数据有同步刷盘和异步刷盘两种机制,同步刷盘是数据写道磁盘后，返回确认值，异步刷盘是先返回在把数据写入磁盘

e.读写性能

利用linux的sendfile机制，将消息内容直接输出到sokect管道，避免系统调用

采用零拷贝技术，数据操作很快

零拷贝:主要是从文件的读取，写入，传输方面考虑，减少了数据的复制，可以参考下篇文章

https://www.linuxjournal.com/article/6345

客户端寻址方式

1.代码指定

Producer.setNamesrvAddr(“192.168.122.1:9876”);

或

Consumer.setNamesrvAddr(“192.168.122.1:9876”);

2.Java启动时指定参数

-Drocketmq.namesrv.addr=192.168.122.1:9876

3.环境变量指定NameServe地址

export NAMESRV_ADDR=192.168.122.1:987

4.http静态服务器寻址

客户端启动后，会定时访问一个http静态服务器:http://jmenv.tbsite.net:8080/rocketmq/msaddr，该服务响应nameserver地址

客户端默认每隔2分钟访问一次这个HTTP服务器，并更新本地的NameServer地址。URL已经在代码中写死，可通过修改/etc/hosts文件来改变要访问的服务器，例如在/etc/hosts增加如下配置：

10.232.22.67     jmenv.tbsite.net

猜你喜欢

转载自blog.csdn.net/strong_yu/article/details/83308146
今日推荐
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
周排行
rbac——界面、权限
Apache CXF + SpringMVC 整合发布WebService
so插件化
Vue.js实战系列---图标字体制作（svg格式）
PAT乙级 1007 素数对猜想(孪生素数对) (20分) ---（C语言 + 详细注释）
被IRM保护的文档，打开失败
Calendar和Date计算日期差的小问题
win10子系统ubuntu18.4安装docker
利用Wrap Shell Script定位Android Native内存泄漏
MySQL: Transaction (Part I - Basic Concept)
每日归档
更多
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022