Redis集群搭建(基于6.x)

一、CentOS安装依赖

1. centos 系统版本:

[root@dn3 redis-6.0.8]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.7.1908 (Core)
Release:        7.7.1908
Codename:       Core

2. 下载redis源码安装包

# 下载redis安装包
wget -c http://download.redis.io/releases/redis-6.0.8.tar.gz

3. 安装redis依赖库

# 安装redis依赖组件
yum -y install gcc
yum -y install centos-release-scl
yum -y install devtoolset-9-gcc devtoolset-9-gcc-c++ devtoolset-9-binutils
scl enable devtoolset-9 bash
# 启动gcc
echo "source /opt/rh/devtoolset-9/enable" >>/etc/profile
yum -y install tcl

4. 安装redis

cd /opt/software

# 解压并编译redis源码包
tar xzf redis-6.0.8.tar.gz -C /opt/module

cd redis-6.0.8

make PREFIX=/usr/local/redis install

5. Standalone Mode启动redis测试

# 启动redis server
[root@dn3 redis-6.0.8]# src/redis-server 
16109:C 24 Jul 2022 16:43:23.076 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16109:C 24 Jul 2022 16:43:23.076 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=16109, just started
16109:C 24 Jul 2022 16:43:23.076 # Warning: no config file specified, using the default config. In order to specify a config file use src/redis-server /path/to/redis.conf
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 6.0.8 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 16109
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

16109:M 24 Jul 2022 16:43:23.078 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
16109:M 24 Jul 2022 16:43:23.078 # Server initialized
16109:M 24 Jul 2022 16:43:23.078 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
16109:M 24 Jul 2022 16:43:23.078 * Ready to accept connections

# 这种启动方式不好,手动执行Ctrl+c关闭
^C16109:signal-handler (1658653088) Received SIGINT scheduling shutdown...
16109:M 24 Jul 2022 16:58:08.970 # User requested shutdown...
16109:M 24 Jul 2022 16:58:08.970 * Saving the final RDB snapshot before exiting.
16109:M 24 Jul 2022 16:58:08.975 * DB saved on disk
16109:M 24 Jul 2022 16:58:08.975 # Redis is now ready to exit, bye bye...

Redis daemonize参数说明:
A、redis.conf配置文件中daemonize守护线程，默认是NO。
B、daemonize是用来指定redis是否要用守护线程的方式启动。

daemonize 设置yes或者no区别
daemonize:yes:redis采用的是单进程多线程的模式。当redis.conf中选项daemonize设置成yes时，代表开启守护进程模式。在该模式下，redis会在后台运行，并将进程pid号写入至redis.conf选项pidfile设置的文件中，此时redis将一直运行，除非手动kill该进程。
daemonize:no: 当daemonize选项设置成no时，当前界面将进入redis的命令行界面，exit强制退出或者关闭连接工具(putty,xshell等)都会导致redis进程退出。

Standalone Mode 后台方式运行:

[root@dn3 redis-6.0.8]# /opt/module/redis-6.0.8/src/redis-server /opt/module/redis-6.0.8/redis.conf 
[root@dn3 redis-6.0.8]# 

# 进入客户端
[root@dn3 redis-6.0.8]# /usr/local/redis/bin/redis-cli -h localhost -p 6379
localhost:6379> keys *
(empty array)

# 插入数据后,再次查询
localhost:6379> keys *
1) "DIM:DIM_BASE_CATEGORY1:19"
localhost:6379> get "DIM:DIM_BASE_CATEGORY1:19"
"{
    
    \"ID\":\"19\",\"NAME\":\"ECHO\"}"
localhost:6379> 

# 去除json中的转义字符(redis-cli命令后添加--raw参数)
[root@dn3 redis-6.0.8]# /usr/local/redis/bin/redis-cli -h localhost -p 6379 --raw
localhost:6379> keys *
DIM:DIM_BASE_CATEGORY1:19
localhost:6379> get "DIM:DIM_BASE_CATEGORY1:19"
{
    
    "ID":"19","NAME":"ECHO"}

# 测试ttl
localhost:6379> expire DIM:DIM_BASE_CATEGORY1:19 10
1

localhost:6379> ttl "DIM:DIM_BASE_CATEGORY1:19"
9
localhost:6379> ttl "DIM:DIM_BASE_CATEGORY1:19"
7
localhost:6379> ttl "DIM:DIM_BASE_CATEGORY1:19"
6
localhost:6379> ttl "DIM:DIM_BASE_CATEGORY1:19"
1
localhost:6379> ttl "DIM:DIM_BASE_CATEGORY1:19"
0
localhost:6379> ttl "DIM:DIM_BASE_CATEGORY1:19"
-2
localhost:6379> ttl "DIM:DIM_BASE_CATEGORY1:19"
-2

设置允许远程访问:

• 添加注释(默认关闭): bind 127.0.0.1 ::1
• 添加注释(默认打开): #bind 127.0.0.1
• 保护模式设置为no(默认yes): protected-mode no

################################## NETWORK #####################################

# By default, if no "bind" configuration directive is specified, Redis listens
# for connections from all the network interfaces available on the server.
# It is possible to listen to just one or multiple selected interfaces using
# the "bind" configuration directive, followed by one or more IP addresses.
#
# Examples:
#
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
#
# ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
# internet, binding to all the interfaces is dangerous and will expose the
# instance to everybody on the internet. So by default we uncomment the
# following bind directive, that will force Redis to listen only into
# the IPv4 loopback interface address (this means Redis will be able to
# accept connections only from clients running into the same computer it
# is running).
#
# IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
# JUST COMMENT THE FOLLOWING LINE.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#bind 127.0.0.1

# Protected mode is a layer of security protection, in order to avoid that
# Redis instances left open on the internet are accessed and exploited.
#
# When protected mode is on and if:
#
# 1) The server is not binding explicitly to a set of addresses using the
#    "bind" directive.
# 2) No password is configured.
#
# The server only accepts connections from clients connecting from the
# IPv4 and IPv6 loopback addresses 127.0.0.1 and ::1, and from Unix domain
# sockets.
#
# By default protected mode is enabled. You should disable it only if
# you are sure you want clients from other hosts to connect to Redis
# even if no authentication is configured, nor a specific set of interfaces
# are explicitly listed using the "bind" directive.
protected-mode no

# Accept connections on the specified port, default is 6379 (IANA #815344).
# If port 0 is specified Redis will not listen on a TCP socket.
port 6379

二、分布式集群配置

PROD环境中不会使用standalone mode, 一般均需要配置集群环境.
这里实验3主3从方式的redis集群，需要启动6个redis实例，分别部署在3台服务器上(如果机器数量少, 也可以在一台Server上通过指定不同的端口号来模拟)，每台服务器启动两个redis实例(一主一从)，主实例的端口号为7000，从实例的端口号为7001.

1. 修改redis配置文件

cd /usr/local/redis-6.0.8
# 编辑redis配置文件，修改以下配置信息：
vim redis.conf

# 后台方式运行redis
daemonize yes
# redis server运行端口号，配置文件拷贝到redisCluster文件中后这里的端口号要根据实际情况修改
port 7000
# bind 127.0.0.1 要改成0.0.0.0，不然redis客户端无法通过ip连接服务端
bind 0.0.0.0

# 在 REDIS CLUSTER 配置模块下开启以下配置
# 开启redis集群支持
cluster-enabled yes
# 集群配置文件，redis首次启动时会在redis.conf所在的文件夹下自动创建该文件，注意这里的node-7000.conf要根据实例启动的端口号自行修改
cluster-config-file node-7000.conf
# pidfile的端口号也需要根据实际启动的端口号自行修改
pidfile /var/run/redis_7000.pid
# 请求超时时间
cluster-node-timeout 15000

2.创建redis实例目录

# 创建redis主从实例文件夹
mkdir -p /usr/local/redisCluster/7000
mkdir -p /usr/local/redisCluster/7001

# 拷贝配置文件到redisCluster文件夹中，并各自修改配置文件的port, cluster-config-file 这两个参数
cp /usr/local/redis-6.0.8/redis.conf /usr/local/redisCluster/7000
cp /usr/local/redis-6.0.8/redis.conf /usr/local/redisCluster/7001

# 拷贝redis目录下的bin目录文件到redisCluster文件夹中
cp -r /usr/local/redis/bin/ /usr/local/redisCluster/7000
cp -r /usr/local/redis/bin/ /usr/local/redisCluster/7001

# 将redis-server,  redis-cli拷贝到RedisCluster目录下，便于启动服务和通过客户端连接
cp /usr/local/redis-6.0.8/src/redis-server /usr/local/redisCluster/
cp /usr/local/redis-6.0.8/src/redis-cli /usr/local/redisCluster/

3.构建redis集群启动脚本

分别在每台server上启动redis实例:

添加环境变量

# 将redis添加到系统变量中
vim /etc/profile

export REDIS_HOME=/usr/local/redis
export PATH=$PATH:$REDIS_HOME/bin

# 生效环境变量
source /etc/profile

创建redis集群启动文件
vim /usr/local/redisCluster/start.sh

# 集群启动脚本内容
cd /usr/local/redisCluster
./redis-server /usr/local/redisCluster/7000/redis.conf
./redis-server /usr/local/redisCluster/7001/redis.conf

4.执行redis-cli redis集群启动命令

Redis的实例全部运行之后，redis新版本（5.0以上）可以直接使用redis-cli来创建集群，Redis5.0以下的版本只能通过redis-trib.rb工具来创建集群。

# 创建redis集群，运行命令后会有一个确认的提示，必须输入 yes 才能创建集群
./redis-cli --cluster create dn3:7000 dn3:7001 dn4:7000 dn4:7001 dn5:7000 dn5:7001 --cluster-replicas 1

# 连接redis集群，命令格式：redis-cli -h <ip> -p <port> -a <password>
./redis-cli -h dn3 -p 7000
# 查看当前集群信息
127.0.0.1:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:473
cluster_stats_messages_pong_sent:464
cluster_stats_messages_sent:937
cluster_stats_messages_ping_received:459
cluster_stats_messages_pong_received:473
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:937

# 查看当前集群有多少个节点
127.0.0.1:7000> cluster nodes
c8ac621fcd7a85cb468883bc46e524d349a8c487 dn4:7001@17001 slave 8cde20f19ae9b72c92cbfe6eb5df622a87aa4417 0 1600680450000 1 connected
648728286cc40550017169ce80795ebd72d1805e dn5:7000@17000 master - 0 1600680451376 5 connected 10923-16383
fa24f272828f84d0862665f33b16e292c514944e dn5:7001@17001 slave 5e5bf132bce0fef67cb86d4f0923c317bc9a949c 0 1600680449371 3 connected
5e5bf132bce0fef67cb86d4f0923c317bc9a949c dn4:7000@17000 master - 0 1600680451000 3 connected 5461-10922
8cde20f19ae9b72c92cbfe6eb5df622a87aa4417 dn3:7000@17000 myself,master - 0 1600680450000 1 connected 0-5460
85835667da45edea2a74bae3d0c817a8681fcfe1 dn3:7001@17001 slave 648728286cc40550017169ce80795ebd72d1805e 0 1600680452378 5 connected

5.设置集群连接密码

在所有redis服务器上执行以下命令:

config set masterauth 123456
config set requirepass 123456

执行完命令之后会自动在redis.conf文件末尾添加
这两个密码参数配置，无需重启redis服务即可完成密码修改；
注意：如果是redis集群要保证所有服务的密码都是一样的，否则跨服务连接会出现异常；

# 如果对集群设置密码，requirepass和masterauth都需要设置，否则发生主从切换时，就会遇到授权问题
# 连接命名格式：./redis-cli -c -p <port> -a <password>
## 示例: ./redis-cli -h 10.0.27.38 -c -p 7000 -a 123456
127.0.0.1:7000> config set masterauth 123456
OK
127.0.0.1:7000> config set requirepass 123456
OK
127.0.0.1:7000> config rewrite
OK

三、redis集群运维

1.设置redis开机自启动

cd /etc/init.d/

vim redis.sh

#!/bin/sh
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
# chkconfig: 2345 80 90
# description: redis start up script

REDISPORT0=7000
REDISPORT1=7001
EXEC=/usr/local/redisCluster/redis-server
CLIEXEC=/usr/local/redisCluster/redis-cli

case "$1" in
    start)
        CONF="/usr/local/redisCluster/${REDISPORT0}/redis.conf"
        echo "Starting Redis server on $REDISPORT0..."
        $EXEC $CONF &
!
        CONF="/usr/local/redisCluster/${REDISPORT1}/redis.conf"
        echo "Starting Redis server on $REDISPORT1..."
        $EXEC $CONF &
!
        ;;
stop)
    es_pid=`ps -aux|grep redis|grep ${
     
     REDISPORT0}|grep -v grep|awk '{print $2}'`
    kill -9 $es_pid
    echo "$REDISPORT0 stopped"
    es_pid=`ps -aux|grep redis|grep ${
     
     REDISPORT1}|grep -v grep|awk '{print $2}'`
    kill -9 $es_pid
    echo "$REDISPORT1 stopped"
    ;;
*)
    echo "start|stop"
    ;;
esac

exit $?

2. 给脚本添加可执行权限&加入开机启动服务

# 给启动脚本添加可执行权限
chmod +x /etc/init.d/redis
# 注册为系统服务
chkconfig --add redis
# 列出名字为redis的系统服务
chkconfig --list redis
# 设置为开机运行服务
chkconfig redis on

四、安装中遇到的问题

1. 连接 redis集群报错：(error) CLUSTERDOWN The cluster is down

运行 cluster info 命令发现节点的状态是fail的（不是所有的节点状态都是fail的，只有异常的节点运行命令状态才是fail，其他正常的节点运行命令返回的状态是OK，所有要排查所有的节点）

解决方案：
如果redis中没有重要数据，可以重新创建redis集群。

重新创建步骤：

• 1.可以先停止所有的redis服务；
• 2.删除所有节点下的nodes-7000.conf（该文件的名字是根据redis.config配置文件中的cluster-config-file配置设置的），dump.rdb两个文件；
• 3.重新启动所有redis节点服务，重新创建集群: ./redis-cli --cluster create dn3:7000 dn3:7001 dn4:7000 dn4:7001 dn5:7000 dn5:7001 --cluster-replicas 1 -a