Centos7&Docker builds Rocketmq environment

Method 1: Centos7 method

1. The required pre-environment

2. Pre-installation settings
1. Install jdk 1.8 (steps omitted)
Download and decompress the installation package to the corresponding directory (/usr/local/java/jdk1.8.0_211/bin/java),

Just configure the environment variable:

#打开环境变量配置文件
vim /etc/profile

########### profile 文件中新增如下三行 ###########
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64
export CLASS_PATH="$JAVA_HOME/lib"
export PATH="$PATH:$JAVA_HOME/bin"

# 配置立刻生效
source /etc/profile

2. Install maven

(1), download, download *.tar.gz

wget https://downloads.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz

(2), unzip, rename, and move to the installation directory you want to place

# 解压
tar -zxvf apache-maven-3.8.4-bin.tar.gz 

# 重命名
mv apache-maven-3.8.4 maven

# 移动maven到/usr/local/
mv maven /usr/local/

# 修改maven配置文件,添加阿里云仓库
    <mirror> 
         <id>alimaven</id> 
         <name>aliyun maven</name> 
         <url>https://maven.aliyun.com/nexus/content/groups/public/</url> 
         <mirrorOf>central</mirrorOf> 
     </mirror>
# 如果jar拉取不下来,可以换成maven仓库
    <mirror> 
         <id>maven2</id> 
         <name>maven2</name> 
         <url>https://repo1.maven.org/maven2/</url> 
         <mirrorOf>central</mirrorOf> 
     </mirror>

# 配置环境变量
vim /etc/profile

# profile 文件中新增如下两行
export  MAVEN_HOME=/usr/local/maven
export PATH=$PATH:$MAVEN_HOME/bin

# 配置立刻生效
source /etc/profile

# 检验是否安装成功
mvn -v

3. Install RocketMQ

1. Download

Download from github (download zip package)

Source code zip source code package: source code package (4.9.2 version)
download extension package (from the original rocketmq-console, separate independent project): extension package (page console)
2. Start mqnamesrv service

# 解压源码包
unzip rocketmq-rocketmq-all-4.9.2.zip
# 进入解压的源码包
cd rocketmq-rocketmq-all-4.9.2
# 编译源码
mvn -Prelease-all -DskipTests clean install -U
# 找到编译后的文件包(distribution/target/rocketmq-4.9.2)
cd distribution/target
# 将rocketmq-4.9.2重新命名为rocketmq并且移动到/usr/local/下
mv rocketmq-4.9.2 rocketmq
mv rocketmq /usr/local/
# 进入bin目录并启动mqnamesrv服务(如果启动失败,可打开runserver.sh,把最小需要内存4G调小)
cd /usr/local/rocketmq/bin
./mqnamesrv

3. Start the mqbroker service

# 重新打开shell窗口,启动mqbroker服务
cd /usr/local/rocketmq/bin
./mqbroker

Note: If the following error is reported at startup, it means that the memory is insufficient, and the minimum memory required in the adjustable runbroker.sh is
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000005c0000000, 8589934592, 0) failed ; error='Cannot allocate memory' (errno=12)

View the runbroker.sh file, you can see that at least 8g memory is required to start mqbroker

JAVA_OPT="${JAVA_OPT} -server -Xms8g -Xmx8g"

solution:

# 进入runbroker.sh
vim runbroker.sh
# 修改最小需要启动内存大小为
JAVA_OPT="${JAVA_OPT} -server -Xms512m -Xmx512m"
# 重新启动mqbroker服务,并且连接到注册中心,否则控制台找不到broker
./mqbroker -n localhost:9876

如果找不到主机，可修改host添加一条对应
vim /etc/hosts
如果需要支持sql过滤需要在，broker.conf配置文件里面增加一条配置
enablePropertyFilter=true
同时启动的时候需要加上配置文件(conf目录)
../bin/mqbroker -n localhost:9876 -c broker.conf

4. Add test configuration

添加mqnamesrv环境变量,尽量添加到export一起
vi tools.sh
export NAMESRV_ADDR=localhost:9876

测试消息生产消费：
生产: 
./tools.sh org.apache.rocketmq.example.quickstart.Producer
消费:
./tools.sh org.apache.rocketmq.example.quickstart.Consumer

5. Start the extended service databoard and web console

# 解压下载的扩展文件rocketmq-dashboard-master.zip
unzip rocketmq-dashboard-master.zip
# 进入解压文件，并进入到rocketmq-console
cd rocketmq-dashboard-master
# 控制台为springboot项目,修改application.properties配置,设置注册中心地址
rocketmq.config.namesrvAddr=127.0.0.1:9876

# 执行编译
mvn clean package -Dmaven.test.skip=true
# 找到编译后的rocketmq-dashboard-1.0.1-SNAPSHOT.jar
#并复制到/usr/local/rocketmq-dashboard(需要新建rocketmq-dashboard)
cd target
cp rocketmq-dashboard-1.0.1-SNAPSHOT.jar /usr/local/rocketmq-dashboard/
# 进入 /usr/local/rocketmq-dashboard/,启动web服务
cd /usr/local/rocketmq-dashboard/
java -jar rocketmq-dashboard-1.0.1-SNAPSHOT.jar
# 如果要后台启动,使用
nohup java -jar rocketmq-dashboard-1.0.1-SNAPSHOT.jar

Problems encountered:

[INFO] Downloading https://nodejs.org/dist/v16.2.0/node-v16.2.0-linux-x64.tar.gz to /root/.m2/repository/com/github/eirslett/node/16.2.0/node-16.2.0-linux-x64.tar.gz
[INFO] No proxies configured
[INFO] No proxy was configured, downloading directly

It is possible that due to network reasons, when compiling and downloading node-v16.2.0-linux-x64.tar.gz, it will fail to download. At this time, you can manually download https://nodejs.org/dist/v16.2.0/node -v16.2.0-linux-x64.tar.gz , then upload it to the service, and place it in the corresponding location /root/.m2/repository/com/github/eirslett/node/16.2.0/, and put the file The name is changed from node-v16.2.0-linux-x64.tar.gz to node-16.2.0-linux-x64.tar.g, which meets the compilation requirements, then recompile and perform the above steps

After the service is started, the browser can access the web console, ip: 8080 (default port 8080)

If after startup, the startup log reports an error
Caused by: org.apache.rocketmq.remoting.exception.RemotingConnectException: connect to null failed

Solution:

# 找到broker.conf 文件(rocketmq/conf/broker.conf),新增如下两行
namesrvAddr=127.0.0.1:9876
# 指定的ip地址即是linux本身的IP地址
brokerIP1=10.118.241.145

# 重新启动mqnamesrv和mqbroker以及rocketmq-console服务
./mqnamesrv    
./mqbroker -n localhost:9876 -c ../conf/broker.conf
java -jar rocketmq-dashboard-1.0.1-SNAPSHOT.jar

# 后台启动方式
./mqnamesrv &    
./mqbroker -n localhost:9876 -c ../conf/broker.conf &
nohup java -jar rocketmq-dashboard-1.0.1-SNAPSHOT.jar --rocketmq.config.namesrvAddr=localhost:9876 > console-ng.log 2>&1&

Four, the pit encountered

Before starting the test program sent by the official website to send messages, you need to enable the default service port number (9876) and the VIP port number of rocketmq, otherwise, an error will be reported when sending messages

Vip port number viewing method, open the rocketmq-console control page

Open the response port number

firewall-cmd --zone=public --add-port=9876/tcp --permanent
firewall-cmd --zone=public --add-port=10911/tcp --permanent
firewall-cmd --reload

After the configuration is complete, no error is reported after restarting the console. I went to find an official website producer example to send a message. As a result, because the virtual machine is used, the disk is relatively small, and the utilization rate reaches 91%. The program reports an error, which means that the disk size is insufficient.

Caused by: org.apache.rocketmq.client.exception.MQBrokerException: CODE: 14  DESC: service not available now. It may be caused by one of the following reasons: the broker's disk is full [CL:  0.91 CQ:  0.91 INDEX:  0.91], messages are put to the slave, message store has been shut down, etc. BROKER: 10.118.241.145:10911

By default, rocketmq will treat the ratio of remaining disks as less than 75% (this ratio seems to be different in different versions of rocketmq) as insufficient disk space.

Solution:

1. Expand the disk

2. Set borker.conf to increase the threshold for reporting insufficient disk

vim broker.conf
# 在文件新增一行配置，磁盘使用率达到99%才允许报磁盘不足
diskMaxUsedSpaceRatio=99

Then, restart the broker service, send the message again, and send it successfully
Note: When the producer sends the message code, do not call shutdown after sending the message, otherwise the web console will not be able to query the producer page

# 生产环境一定要注释掉
//producer.shutdown();

At this point, the entire simple construction process is completed

Method 2: Docker method

1. Pull the image to create the namesrv service

docker pull rocketmqinc/rocketmq

Create namesrv data storage path

mkdir -p  /data/docker/rocketmq/data/namesrv/logs   /data/docker/rocketmq/data/namesrv/store

build container

docker run -d \
--restart=always \
--name rmqnamesrv \
-p 9876:9876 \
-v /data/docker/rocketmq/data/namesrv/logs:/root/logs \
-v /data/docker/rocketmq/data/namesrv/store:/root/store \
-e "MAX_POSSIBLE_HEAP=100000000" \
rocketmqinc/rocketmq \
sh mqnamesrv

2. Create a Broker node

Create broker data storage path

mkdir -p  /data/docker/rocketmq/data/broker/logs /data/docker/rocketmq/data/broker/store /data/docker/rocketmq/conf

Create a broker to read the configuration file

vi /data/docker/rocketmq/conf/broker.conf

The content of the configuration file is as follows:

# 所属集群名称，如果节点较多可以配置多个
brokerClusterName = DefaultCluster
#broker名称，master和slave使用相同的名称，表明他们的主从关系
brokerName = broker-a
#0表示Master，大于0表示不同的slave
brokerId = 0
#表示几点做消息删除动作，默认是凌晨4点
deleteWhen = 04
#在磁盘上保留消息的时长，单位是小时
fileReservedTime = 48
#有三个值：SYNC_MASTER，ASYNC_MASTER，SLAVE；同步和异步表示Master和Slave之间同步数据的机制；
brokerRole = ASYNC_MASTER
#刷盘策略，取值为：ASYNC_FLUSH，SYNC_FLUSH表示同步刷盘和异步刷盘；SYNC_FLUSH消息写入磁盘后才返回成功状态，ASYNC_FLUSH不需要；
flushDiskType = ASYNC_FLUSH
# 设置broker节点所在服务器的ip地址
brokerIP1 = 192.168.2.115
# 开启权限认证
#aclEnable=true

To enable permission authentication, you need to create plain_acl.yml

vi /data/docker/rocketmq/conf/plain_acl.yml

Configure user permissions

globalWhiteRemoteAddresses:
accounts:
- accessKey: RocketMQ
  secretKey: 123456
  whiteRemoteAddress:
  admin: false
  defaultTopicPerm: DENY
  defaultGroupPerm: SUB
  topicPerms:
  # 用户RocketMQ只能发送TopicTest的消息，其他topic无权限发送
  - TopicTest=PUB
  groupPerms:
  # the group should convert to retry topic
  - oms_consumer_group=DENY
- accessKey: admin
  secretKey: 123456
  whiteRemoteAddress:
  # if it is admin, it could access all resources
  admin: true

Build a broker container

Non-open permission method: (recommended)

docker run -d  \
--restart=always \
--name rmqbroker \
--link rmqnamesrv:namesrv \
-p 10911:10911 \
-p 10909:10909 \
-v /data/docker/rocketmq/data/broker/logs:/root/logs \
-v /data/docker/rocketmq/data/broker/store:/root/store \
-v /data/docker/rocketmq/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf \
-e "NAMESRV_ADDR=namesrv:9876" \
-e "MAX_POSSIBLE_HEAP=200000000" \
rocketmqinc/rocketmq \
sh mqbroker -c /opt/rocketmq-4.4.0/conf/broker.conf

How to enable permissions:

docker run -d  \
--restart=always \
--name rmqbroker \
--link rmqnamesrv:namesrv \
-p 10911:10911 \
-p 10909:10909 \
-v /data/docker/rocketmq/data/broker/logs:/root/logs \
-v /data/docker/rocketmq/data/broker/store:/root/store \
-v /data/docker/rocketmq/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf \
-v /data/docker/rocketmq/conf/plain_acl.yml:/opt/rocketmq-4.4.0/conf/plain_acl.yml \
-e "NAMESRV_ADDR=namesrv:9876" \
-e "MAX_POSSIBLE_HEAP=200000000" \
rocketmqinc/rocketmq \
sh mqbroker -c /opt/rocketmq-4.4.0/conf/broker.conf

3. Create rockermq-console service

pull image

docker pull pangliang/rocketmq-console-ng

Build the rockermq-console-ng container

docker run -d \
--restart=always \
--name rmqadmin \
-e "JAVA_OPTS=-Drocketmq.namesrv.addr=192.168.2.115:9876 \
-Dcom.rocketmq.sendMessageWithVIPChannel=false" \
-p 9999:8080 \
pangliang/rocketmq-console-ng

Access: ip:9999 to access the console

If you need to open the management page to log in, create a users.properties file in the /data/docker/rocketmq/dashboard folder, and update the username/password to take effect without restarting

# This file supports hot change, any change will be auto-reloaded without Console restarting.
# Format: a user per line, username=password[,N] #N is optional, 0 (Normal User); 1 (Admin)
 
# Define Admin
# =============用户名和密码规则「用户名=密码,权限」，这里的权限为1表示管理员，为0表示普通用户=============
# 例如：admin=admin123,1
admin=admin,1
user=user,0
 
 
# Define Users
# =============屏蔽下边两个账户=============
#user1=user1
#user2=user2

First delete the enabled startup dashboard instance, then restart to create the container

docker run -d \
--restart=always \
--name rmqadmin -v /data/docker/rocketmq/dashboard:/tmp/rocketmq-console/data \
-e "JAVA_OPTS=-Drocketmq.namesrv.addr=192.168.2.115:9876 \
-Dcom.rocketmq.sendMessageWithVIPChannel=false \
-Drocketmq.config.loginRequired=true" \
-p 9999:8080 \
pangliang/rocketmq-console-ng

After the container is started, you need to log in to access the console:

At this point, the centos7 build and docker build rocketmq tutorials are over.

Centos7&Docker builds Rocketmq environment

Guess you like