redis - 搭建多master，主从复制+高可用集群

redis cluster集群，要求至少3个master，去组成一个高可用，健壮的分布式的集群，每个master给一个slave，3个master，3个slave，最少的要求

1. 部署目录

/etc/redis（存放redis的配置文件），
/var/redis/对应端口，如：/var/redis/7001；/var/redis/7002（存放redis的持久化文件）

2. 编写配置文件

机器：
192.168.43.170;一个master一个slave; 端口分别为7001和7002; 配置文件为/etc/redis/7001.conf & 7002.conf;
192.168.43.171;一个master一个slave; 端口分别为7003和7004；配置文件为/etc/redis/7003.conf & 7004.conf;
192.168.43.172;一个master一个slave; 端口分别为7005和7006；配置文件为/etc/redis/7005.conf & 7006.conf;

cluster-config-file文件分别为：
/etc/redis-cluster/node-7001.conf & node-7002.conf & …

cluster-config-file文件，是供cluster模式下的redis实例将集群状态保存在那里，包括集群中其他机器的信息，比如节点的上线和下限，故障转移，不是我们去维护的，给它指定一个文件，让redis自己去维护的

按这样的配置方式，如170机器的配置为：

port 7001
cluster-enabled yes
cluster-config-file /etc/redis-cluster/node-7001.conf
cluster-node-timeout 15000
daemonize   yes                         
pidfile     /var/run/redis_7001.pid                         
dir         /var/redis/7001
logfile /var/log/redis/7001.log
bind 192.168.43.170 
appendonly yes

170的7001端口的redis搭建

1.创建目录：

mkdir -p /etc/redis-cluster
mkdir -p /var/log/redis
mkdir -p /var/redis/7001
mkdir -p /var/redis/7002

2.拷贝配置文件:

cp /usr/local/redis-3.2.8/redis.conf /etc/redis/7001.conf
cp /usr/local/redis-3.2.8/redis.conf /etc/redis/7001.conf

修改/etc/redis/7001.conf：

cluster-enabled yes
cluster-config-file /etc/redis-cluster/node-7001.conf
cluster-node-timeout 15000
bind 192.168.43.170
protected-mode no
port 7001
daemonize yes
pidfile /var/run/redis_7001.pid
logfile "/var/log/redis/7001.log"
dir /var/redis/7001
appendonly yes

3.修改配置启动脚本：

cp /usr/local/redis-3.2.8/utils/redis_init_script /etc/init.d/redis_7001

修改redis_7001启动脚本

(1)将端口从6379改为7001

(2)增加HOST_IP=`hostname -i`

(3)修改关闭redis服务的命令，$CLIEXEC -h $HOST_IP -p $REDISPORT shutdown

170的7002端口的redis搭建

修改/etc/redis/7002.conf

cluster-enabled yes
cluster-config-file /etc/redis-cluster/node-7002.conf
cluster-node-timeout 15000
bind 192.168.43.170
protected-mode no
port 7002
daemonize yes
pidfile /var/run/redis_7002.pid
logfile "/var/log/redis/7002.log"
dir /var/redis/7002
appendonly yes

3. 修改配置启动脚本：

cp /usr/local/redis-3.2.8/utils/redis_init_script /etc/init.d/redis_7002

(1)并将端口从6379改为7002

(2)HOST_IP=`hostname -i`

(3)修改关闭redis服务的命令，$CLIEXEC -h$HOST_IP -p $REDISPORT shutdown

为7001和7002创建启动的软连接，方便启动：

cd 
ln -s /etc/init.d/redis_7001 redis_7001
ln -s /etc/init.d/redis_7002 redis_7002

通过命令，分别启动7001和7002端口

[root@eshop-cache01 ~]# pwd
/root
[root@eshop-cache01 ~]# ls
bak.redis_control.sh  redis_7001  redis_7002  redis.conf  redis_control.sh  redis_dump  sbin  sentinel.conf  soft
[root@eshop-cache01 ~]# ./redis_7001 start
Starting Redis server...
[root@eshop-cache01 ~]# ./redis_7002 start
Starting Redis server...
[root@eshop-cache01 ~]# ps -ef | grep redis
root     30579     1  0 07:57 ?        00:00:00 /usr/local/bin/redis-server 192.168.43.170:7001 [cluster]
root     30585     1  0 07:57 ?        00:00:00 /usr/local/bin/redis-server 192.168.43.170:7002 [cluster]
root     30589 30380  0 07:57 pts/0    00:00:00 grep redis

机器170配置并启动完成，机器171和172同理配置，将170的配置文件scp到171和172，稍微做些修改即可完成配置；

将170的配置文件上传，可下载参考：

/etc/rc.d/init.d/redis_7001
/etc/redis/7001.conf

点击下载

至此：170，171，172配置和启动完成：

[root@eshop-cache02 ~]# ps -ef | grep redis
root       413     1  0 15:51 ?        00:00:00 /usr/local/bin/redis-server 192.168.43.171:7003 [cluster]
root       419     1  0 15:51 ?        00:00:00 /usr/local/bin/redis-server 192.168.43.171:7004 [cluster]

[root@eshop-cache03 ~]# ps -ef | grep redis
root     19592     1  0 15:55 ?        00:00:00 /usr/local/bin/redis-server 192.168.43.172:7005 [cluster]
root     19598     1  0 15:56 ?        00:00:00 /usr/local/bin/redis-server 192.168.43.172:7006 [cluster]

4.安装ruby和gem

在172机器上安装ruby和rubygems
安装过程中遇到些问题，参考：
(1) ruby cannot load such file – zlib
(2) redis make: * No rule to make target /include/ruby.h', needed byzlib.o’. Stop.

172机器安装的ruby是：ruby-2.5.1.tar.gz版本的

ruby

tar -zxvf ruby-2.3.1.tar.gz
./configure -prefix=/usr/local/ruby
make && make install
cd /usr/local/ruby
cp bin/ruby /usr/local/bin
cp bin/gem /usr/local/bin

gem
redis-3.3.0.gem

gem install -l ./redis-3.3.0.gem

[root@eshop-cache03 ~]# gem list | grep redis
redis (3.3.0)

5.创建集群

先把redis-trib.rb拷贝到/usr/local/bin
cp /usr/local/redis-3.2.8/src/redis-trib.rb /usr/local/bin

创建集群命令：redis-trib.rb create --replicas 1 192.168.43.170:7001 192.168.43.170:7002 192.168.43.171:7003 192.168.43
.171:7004 192.168.43.172:7005 192.168.43.172:7006

--replicas: 每个master有几个slave
3台机器6个实例，3个master，3个slave

直接把创建集群的日志贴上来观察：
[root@eshop-cache03 local]# redis-trib.rb create --replicas 1 192.168.43.170:7001 192.168.43.170:7002 192.168.43.171:7003 192.168.43
.171:7004 192.168.43.172:7005 192.168.43.172:7006
>>> Creating cluster
/usr/local/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis/client.rb:459: warning: constant ::Fixnum is deprecated
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.43.170:7001
192.168.43.171:7003
192.168.43.172:7005
Adding replica 192.168.43.171:7004 to 192.168.43.170:7001 主备在不同机器上
Adding replica 192.168.43.170:7002 to 192.168.43.171:7003
Adding replica 192.168.43.172:7006 to 192.168.43.172:7005
M: 8594e28845903475c0f8981f891bc7a8f52bb3e5 192.168.43.170:7001
slots:0-5460 (5461 slots) master
S: dbd41e204973fda45124845c8606c0f33f444184 192.168.43.170:7002
replicates d6dfe64b9fa0b8014b73bd59d39ed11c873246d5
M: d6dfe64b9fa0b8014b73bd59d39ed11c873246d5 192.168.43.171:7003
slots:5461-10922 (5462 slots) master
S: 4051da8468ba4671e28aa50465826bab426827b1 192.168.43.171:7004
replicates 8594e28845903475c0f8981f891bc7a8f52bb3e5
M: 2c5f1ad11ca35ed62f2cbf71cf5bd8dac46d98ca 192.168.43.172:7005
slots:10923-16383 (5461 slots) master
S: 6bda4820edcbf26099b725884e9d528d8ff1c4cb 192.168.43.172:7006
replicates 2c5f1ad11ca35ed62f2cbf71cf5bd8dac46d98ca
Can I set the above configuration? (type ‘yes’ to accept): yes 输入yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join…..
>>> Performing Cluster Check (using node 192.168.43.170:7001)
M: 8594e28845903475c0f8981f891bc7a8f52bb3e5 192.168.43.170:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: d6dfe64b9fa0b8014b73bd59d39ed11c873246d5 192.168.43.171:7003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 2c5f1ad11ca35ed62f2cbf71cf5bd8dac46d98ca 192.168.43.172:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: dbd41e204973fda45124845c8606c0f33f444184 192.168.43.170:7002
slots: (0 slots) slave
replicates d6dfe64b9fa0b8014b73bd59d39ed11c873246d5
S: 6bda4820edcbf26099b725884e9d528d8ff1c4cb 192.168.43.172:7006
slots: (0 slots) slave
replicates 2c5f1ad11ca35ed62f2cbf71cf5bd8dac46d98ca
S: 4051da8468ba4671e28aa50465826bab426827b1 192.168.43.171:7004
slots: (0 slots) slave
replicates 8594e28845903475c0f8981f891bc7a8f52bb3e5
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.
[root@eshop-cache03 local]#

检查

命令：redis-trib.rb check 192.168.43.170:7001
[root@eshop-cache03 local]# redis-trib.rb check 192.168.43.170:7001
/usr/local/ruby/lib/ruby/gems/2.5.0/gems/redis-3.3.0/lib/redis/client.rb:459: warning: constant ::Fixnum is deprecated
>>> Performing Cluster Check (using node 192.168.43.170:7001)
M: 8594e28845903475c0f8981f891bc7a8f52bb3e5 192.168.43.170:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: d6dfe64b9fa0b8014b73bd59d39ed11c873246d5 192.168.43.171:7003
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 2c5f1ad11ca35ed62f2cbf71cf5bd8dac46d98ca 192.168.43.172:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: dbd41e204973fda45124845c8606c0f33f444184 192.168.43.170:7002
slots: (0 slots) slave
replicates d6dfe64b9fa0b8014b73bd59d39ed11c873246d5
S: 6bda4820edcbf26099b725884e9d528d8ff1c4cb 192.168.43.172:7006
slots: (0 slots) slave
replicates 2c5f1ad11ca35ed62f2cbf71cf5bd8dac46d98ca
S: 4051da8468ba4671e28aa50465826bab426827b1 192.168.43.171:7004
slots: (0 slots) slave
replicates 8594e28845903475c0f8981f891bc7a8f52bb3e5
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.
[root@eshop-cache03 local]#

感谢龙果