Redis Cluster搭建手册–脚本搭建
实验环境
操作系统: Ubuntu 18.04.1 LTS
redis版本: redis 5.0.2
准备节点
Redis cluster一般要求至少拥有6个节点才能保证其完整的高可用性。需要准备conf,data,log存放路径,建议使用统一目录。
配置conf:
准备conf存放路径
root@sbc-VirtualBox:/etc/redis# mkdir -p /etc/redis/conf
创建conf文件:
root@sbc-VirtualBox:/etc/redis# vim /etc/redis/conf/redis-6379.conf
port 6379
daemonize yes
logfile "6379.log"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6379.conf"
其中:
- port:表示节点端口
- daemonize:表示后台启动
- cluster-enabled:表示开启集群模式
- cluster-node-timeout:表示节点超时时间,单位为毫秒
- cluster-config-file:表示集群内部配置文件
以同样的方式配置6380,6381,6382,6383,6384节点
启动所有节点
按顺序启动节点:
root@sbc-VirtualBox:~# redis-server /etc/redis/conf/redis-6379.conf
root@sbc-VirtualBox:~# for((i=0;i<=4;i++)); do redis-server /etc/redis/conf/redis-638$i.conf;done
确认所有节点状态:
root@sbc-VirtualBox:~# ps -ef|grep redis
root 1709 1 0 11:01 ? 00:00:00 redis-server *:6379 [cluster]
root 1728 1 0 11:02 ? 00:00:00 redis-server *:6380 [cluster]
root 1730 1 0 11:02 ? 00:00:00 redis-server *:6381 [cluster]
root 1735 1 0 11:02 ? 00:00:00 redis-server *:6382 [cluster]
root 1737 1 0 11:02 ? 00:00:00 redis-server *:6383 [cluster]
root 1745 1 0 11:02 ? 00:00:00 redis-server *:6384 [cluster]
root 1753 1668 0 11:02 pts/0 00:00:00 grep --color=auto redis
确认集群状态:
root@sbc-VirtualBox:~# redis-cli -p 6379 cluster info
cluster_state:fail
cluster_slots_assigned:2
cluster_slots_ok:2
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:1
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
各节点加入集群
redis在5.0版本后取消了redis-trib.rb工具,并将其功能集成到了redis-cli --cluster命令中:
root@sbc-VirtualBox:~# redis-cli --cluster help
Cluster Manager Commands:
create host1:port1 ... hostN:portN
--cluster-replicas <arg>
check host:port
info host:port
fix host:port
reshard host:port
--cluster-from <arg>
--cluster-to <arg>
--cluster-slots <arg>
--cluster-yes
--cluster-timeout <arg>
--cluster-pipeline <arg>
rebalance host:port
--cluster-weight <node1=w1...nodeN=wN>
--cluster-use-empty-masters
--cluster-timeout <arg>
--cluster-simulate
--cluster-pipeline <arg>
--cluster-threshold <arg>
add-node new_host:new_port existing_host:existing_port
--cluster-slave
--cluster-master-id <arg>
del-node host:port node_id
call host:port command arg arg .. arg
set-timeout host:port milliseconds
import host:port
--cluster-from <arg>
--cluster-copy
--cluster-replace
help
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
将6个节点都加入集群:
添加clister-replicas 1参数,将为每个master节点添加一个slave节点
root@sbc-VirtualBox:/etc/redis/conf# redis-cli --cluster create 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:6383 127.0.0.1:6384 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:6382 to 127.0.0.1:6379
Adding replica 127.0.0.1:6383 to 127.0.0.1:6380
Adding replica 127.0.0.1:6384 to 127.0.0.1:6381
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: d95d0c2b93047e982669f02fb53c34e52f346a33 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
M: b6fd982ccd30162028dbdb741dbc2c780b8727f1 127.0.0.1:6380
slots:[858],[866],[5461-10922] (5462 slots) master
M: eb6990adf9fd8b51a4bd7553be92628ee24bd058 127.0.0.1:6381
slots:[858],[866],[10923-16383] (5461 slots) master
S: 99f1c320334c9c3e0a6528b0d417213cd50fff56 127.0.0.1:6382
replicates d95d0c2b93047e982669f02fb53c34e52f346a33
S: 4ba5d52746e54c23621c9cb5934e780e158fbb83 127.0.0.1:6383
replicates b6fd982ccd30162028dbdb741dbc2c780b8727f1
S: 04fb4d20630fdc5a3df3b9b72adf5dafe71ba60f 127.0.0.1:6384
replicates eb6990adf9fd8b51a4bd7553be92628ee24bd058
确认分配正确后输入yes:
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382
slots: (0 slots) slave
replicates cf60c7ba6737de0a80528d0a4ab5edfc7867b95c
S: 19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384
slots: (0 slots) slave
replicates 0766f5afb660e1c4bf4da330eb9ff265f5e05694
M: cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383
slots: (0 slots) slave
replicates af745aab508d71d209e17afa5e9470335f43bc9a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
检查集群状态:
root@sbc-VirtualBox:/etc/redis/conf# redis-cli --cluster check 127.0.0.1:6379
127.0.0.1:6379 (af745aab...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:6381 (cf60c7ba...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:6380 (0766f5af...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382
slots: (0 slots) slave
replicates cf60c7ba6737de0a80528d0a4ab5edfc7867b95c
S: 19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384
slots: (0 slots) slave
replicates 0766f5afb660e1c4bf4da330eb9ff265f5e05694
M: cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383
slots: (0 slots) slave
replicates af745aab508d71d209e17afa5e9470335f43bc9a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@sbc-VirtualBox:/etc/redis/conf# redis-cli --cluster info 127.0.0.1:6379
127.0.0.1:6379 (af745aab...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:6381 (cf60c7ba...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:6380 (0766f5af...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
添加节点
配置6385,6386节点的conf:
root@sbc-VirtualBox:/etc/redis/conf# vim /etc/redis/conf/redis-6385.conf
port 6385
daemonize yes
logfile "6385.log"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6385.conf"
root@sbc-VirtualBox:/etc/redis/conf# vim /etc/redis/conf/redis-6386.conf
port 6386
daemonize yes
logfile "6386.log"
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6386.conf"
启动6385,6386节点:
root@sbc-VirtualBox:/etc/redis/conf# redis-server /etc/redis/conf/redis-6385.conf
root@sbc-VirtualBox:/etc/redis/conf# redis-server /etc/redis/conf/redis-6386.conf
将6385以master身份加入集群:
root@sbc-VirtualBox:/etc/redis/conf# redis-cli --cluster add-node 127.0.0.1:6385 127.0.0.1:6379
>>> Adding node 127.0.0.1:6385 to cluster 127.0.0.1:6379
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382
slots: (0 slots) slave
replicates cf60c7ba6737de0a80528d0a4ab5edfc7867b95c
S: 19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384
slots: (0 slots) slave
replicates 0766f5afb660e1c4bf4da330eb9ff265f5e05694
M: cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383
slots: (0 slots) slave
replicates af745aab508d71d209e17afa5e9470335f43bc9a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:6385 to make it join the cluster.
[OK] New node added correctly.
查看集群node信息:
root@sbc-VirtualBox:/etc/redis/conf# redis-cli -p 6385 cluster nodes
af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379@16379 master - 0 1545895589000 1 connected 0-5460
cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381@16381 master - 0 1545895589994 3 connected 10923-16383
19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384@16384 slave 0766f5afb660e1c4bf4da330eb9ff265f5e05694 0 1545895589000 2 connected
0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380@16380 master - 0 1545895588000 2 connected 5461-10922
6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382@16382 slave cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 0 1545895589000 3 connected
404ec6690d4057ea5a837b46b325ecb973fd75a8 127.0.0.1:6385@16385 myself,master - 0 1545895585000 0 connected
937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383@16383 slave af745aab508d71d209e17afa5e9470335f43bc9a 0 1545895590996 1 connected
将6386以6385的slave身份加入集群:
root@sbc-VirtualBox:/etc/redis/conf# redis-cli --cluster add-node 127.0.0.1:6386 127.0.0.1:6379 --cluster-slave --cluster-master-id 404ec6690d4057ea5a837b46b325ecb973fd75a8
>>> Adding node 127.0.0.1:6386 to cluster 127.0.0.1:6379
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382
slots: (0 slots) slave
replicates cf60c7ba6737de0a80528d0a4ab5edfc7867b95c
M: 404ec6690d4057ea5a837b46b325ecb973fd75a8 127.0.0.1:6385
slots: (0 slots) master
S: 19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384
slots: (0 slots) slave
replicates 0766f5afb660e1c4bf4da330eb9ff265f5e05694
M: cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383
slots: (0 slots) slave
replicates af745aab508d71d209e17afa5e9470335f43bc9a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:6386 to make it join the cluster.
Waiting for the cluster to join
>>> Configure node as replica of 127.0.0.1:6385.
[OK] New node added correctly.
查看状态:
root@sbc-VirtualBox:/etc/redis/conf# redis-cli -p 6385 cluster nodes
af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379@16379 master - 0 1545895685354 1 connected 0-5460
cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381@16381 master - 0 1545895684000 3 connected 10923-16383
19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384@16384 slave 0766f5afb660e1c4bf4da330eb9ff265f5e05694 0 1545895684000 2 connected
0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380@16380 master - 0 1545895683000 2 connected 5461-10922
6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382@16382 slave cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 0 1545895685000 3 connected
404ec6690d4057ea5a837b46b325ecb973fd75a8 127.0.0.1:6385@16385 myself,master - 0 1545895686000 0 connected
937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383@16383 slave af745aab508d71d209e17afa5e9470335f43bc9a 0 1545895687362 1 connected
6e29b219598f00b9e097558835e0b17ee8304271 127.0.0.1:6386@16386 slave 404ec6690d4057ea5a837b46b325ecb973fd75a8 0 1545895686357 0 connected
删除节点
删除slave节点
以6386为例:
查看节点状态
root@sbc-VirtualBox:/etc/redis/conf# redis-cli -p 6385 cluster nodes
af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379@16379 master - 0 1545905327087 1 connected 0-5460
cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381@16381 master - 0 1545905328094 3 connected 10923-16383
19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384@16384 slave 0766f5afb660e1c4bf4da330eb9ff265f5e05694 0 1545905330107 2 connected
0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380@16380 master - 0 1545905327000 2 connected 5461-10922
6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382@16382 slave cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 0 1545905329101 3 connected
404ec6690d4057ea5a837b46b325ecb973fd75a8 127.0.0.1:6385@16385 myself,master - 0 1545905329000 0 connected
937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383@16383 slave af745aab508d71d209e17afa5e9470335f43bc9a 0 1545905328000 1 connected
6e29b219598f00b9e097558835e0b17ee8304271 127.0.0.1:6386@16386 slave 404ec6690d4057ea5a837b46b325ecb973fd75a8 0 1545905328000 0 connected
执行删除节点命令
root@sbc-VirtualBox:/etc/redis/conf# redis-cli --cluster del-node 127.0.0.1:6386 6e29b219598f00b9e097558835e0b17ee8304271
>>> Removing node 6e29b219598f00b9e097558835e0b17ee8304271 from cluster 127.0.0.1:6386
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
删除master节点
以6385为例:
先要执行reshard命令,移除6385上的slot
root@sbc-VirtualBox:/etc/redis/conf# redis-cli -p 6379 --cluster reshard 127.0.0.1:6385
>>> Performing Cluster Check (using node 127.0.0.1:6385)
M: f416336be36b5249a0a665a407385710f93b009a 127.0.0.1:6385
slots: (0 slots) master
S: 19dcd6d144c04f8cdb21003c034951c5d11138f2 127.0.0.1:6384
slots: (0 slots) slave
replicates 0766f5afb660e1c4bf4da330eb9ff265f5e05694
M: cf60c7ba6737de0a80528d0a4ab5edfc7867b95c 127.0.0.1:6381
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 937b3138c7c5e8320f4d2305f5c0095ec006a0c0 127.0.0.1:6383
slots: (0 slots) slave
replicates af745aab508d71d209e17afa5e9470335f43bc9a
M: 0766f5afb660e1c4bf4da330eb9ff265f5e05694 127.0.0.1:6380
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 6b8461ad6c6fb2e3c4dc3fbf01051357e85ddeb8 127.0.0.1:6382
slots: (0 slots) slave
replicates cf60c7ba6737de0a80528d0a4ab5edfc7867b95c
M: af745aab508d71d209e17afa5e9470335f43bc9a 127.0.0.1:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
将6385上的slot全转移给其他master节点,再执行删除节点操作:
root@sbc-VirtualBox:/etc/redis/conf# redis-cli --cluster del-node 127.0.0.1:6385 f416336be36b5249a0a665a407385710f93b009a
>>> Removing node f416336be36b5249a0a665a407385710f93b009a from cluster 127.0.0.1:6385
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.