Docker 安装 Keepalived + Nginx 并实现高可用

文章目录

Docker 安装 Keepalived + Nginx 并实现高可用

准备工作

下载 centos 镜像并配置环境

下载安装 Nginx

配置 yum 源

下载安装 Keepalived

编译安装

配置 Keepalived

检查配置

测试

容器创建快照
创建主从容器
进入 maser 容器修改配置
进入 slave 容器修改配置
测试 Keepalived

需求目标：基于 Centos 镜像安装 Keepalived + Nginx 并实现高可用

准备工作

下载 centos 镜像并配置环境

$ docker pull centos # 拉取最新 centos
$ docker images # 查看镜像
$ docker run -itd --name centos centos # 后台启动一个容器
$ docker exec -it centos bash # 进入容器
$ yum -y upgrade # 升级所有包 
$ yum install -y yum-utils vim make wget gcc openssl openssl-devel popt-devel # 安装相关依赖

下载安装 Nginx

配置 yum 源

官方参考文档 http://nginx.org/en/linux_packages.html

$ cat /etc/yum.repos.d/nginx.repo

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

$ yum install -y nginx # 安装 Nginx
$ whereis nginx # 查看安装路径,默认配置文件在 /etc/nginx
$ nginx -t # 插件配置
$ nginx # 启动
$ curl localhost # 检查启动是否成功

下载安装 Keepalived

Keepalived 官网下载最新版本

$ mkdir -p /mnt/software && cd /mnt/software 
$ wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
$ docker cp keepalived-2.0.20.tar.gz centos:/mnt/ # 下载慢！！也可以从本地 copy
$ tar -zxvf keepalived-2.0.20.tar.gz

编译安装

$ cd keepalived-2.0.20
$ ./configure --prefix=/usr/local/keepalived

编译完后可能会出现以下警告

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

按照提示按照

$ yum -y install libnl libnl-devel
No match for argument: libnl
No match for argument: libnl-devel
Error: Unable to find a match: libnl libnl-devel

what 没有匹配的软件源? 我们只需要用到 VRRP 功能，只要确保 Use VRRP Framework 、Use VRRP VMAC、Use VRRP authentication 为 Yes 即可，当然也可以单独找 RPM 来安装 libnl 。

https://centos.pkgs.org/7/centos-x86_64/libnl-1.1.4-3.el7.x86_64.rpm.html

$ wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libnl-1.1.4-3.el7.x86_64.rpm
$ wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libnl-devel-1.1.4-3.el7.x86_64.rpm
$ rpm -ivh libnl-1.1.4-3.el7.x86_64.rpm
$ rpm -ivh libnl-devel-1.1.4-3.el7.x86_64.rpm
$ ./configure --prefix=/usr/local/keepalived
$ make && make install

配置 Keepalived

$ cp /mnt/software/keepalived-2.0.20/keepalived/etc/init.d/keepalived /etc/init.d/
$ cp /mnt/software/keepalived-2.0.20/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
$ mkdir -p /etc/keepalived
$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
$ chmod +x /etc/init.d/keepalived
$ ip address # 本机 ip 172.17.0.2 网卡 eth0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

默认将 Keepalived 配置文件放在 /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
	 # 负载均衡标识，在局域网内应该是唯一的
   router_id LVS_DEVEL
   enable_script_security
   # 建议新建用户来执行脚本
   script_user root
}

vrrp_script chk_nginx {
    script "/etc/keepalived/check.sh"
    # 每2秒检测一次nginx的运行状态
    interval 2 
    # 失败一次，将自己的优先级调整为-5
    weight -5
    # require 2 failures for KO
    fall 2
    # require 2 successes for OK
    rise 2
}

vrrp_instance VI_1 {
    # 标示状态为 MASTER 备份机为 BACKUP
    state MASTER
    # 设置实例绑定的网卡
    interface eth0
    # 同一实例下 virtual_router_id 必须相同
    virtual_router_id 51
    # MASTER 权重要高于 BACKUP 比如 BACKUP 为99
    priority 100
    # MASTER 与 BACKUP 负载均衡器之间同步检查的时间间隔，单位是秒
    advert_int 1 
    # 设置认证
    authentication { 
    		# 主从服务器验证方式
        auth_type PASS
        auth_pass 1111
    }
    # 虚拟IP，两个节点设置必须一致，可以设置多个
    virtual_ipaddress { 
        172.17.0.200
        172.17.0.210
    }
    track_script {
       chk_nginx
    }
}

$ cd /etc/keepalived
$ touch check.sh
$ chmod +x check.sh

check.sh 脚本内容如下

#!/bin/bash
Nginx=`ps -ef | grep nginx | grep -v grep | wc -l`
echo 'nginx status ' $Nginx
if [ $Nginx -eq 0 ]
then
    echo 'nginx shutdown...'
    nginx
    echo 'start nginx ...'
    sleep 2
    if [ `ps -ef | grep nginx | grep -v grep | wc -l` -eq 0 ]
    then
				ps -ef | grep keepalived | grep -v grep | awk '{print $2}' | xargs kill -9
				echo 'kill keepalived...'
    else
	echo 'nginx started...'
    fi
fi
echo 'script end...'

脚本作用：当 nginx 进程不存在时，会自动重启 nginx 服务，休眠 2 秒再次检查 nginx 进程，如果不存在就停止 keepalived 服务

测试脚本：

$ ps ax|grep nginx
   40 pts/1    S+     0:00 grep --color=auto nginx
$ ./check.sh
nginx status  0
nginx shutdown...
start nginx ...
nginx started...
script end...
$ ps ax|grep nginx
   48 ?        Ss     0:00 nginx: master process nginx
   49 ?        S      0:00 nginx: worker process
   57 pts/1    S+     0:00 grep --color=auto nginx

检查配置

$ keepalived -t

至此，Keepalived 以及 Nginx 都配置完成。

测试

容器创建快照

$ docker ps -a | grep centos
ace32d2fa597 centos "/bin/bash" About an hour ago Up About an hour  centos
$ docker commit ace32d2fa597 centos_keepalived_nginx:0.0.1 # 创建该容器的快照
$ docker images | grep centos_keepalived_nginx
centos_keepalived_nginx   0.0.1               801e75c18232        51 seconds ago      569MB

创建主从容器

根据快照创建主从容器

$ docker run --privileged -itd -p 6698:80 --name keepalived_master --hostname keepalived_master centos_keepalived_nginx:0.0.1 /usr/sbin/init
$ docker run --privileged -itd -p 6699:80 --name  keepalived_slave --hostname keepalived_slave centos_keepalived_nginx:0.0.1 /usr/sbin/init

进入 maser 容器修改配置

$ docker exec -it keepalived_master bash # 进入 master 容器
$ vi /usr/share/nginx/html/index.html # 任意修改一点内容与 slave 做区分
$ nginx
$ curl localhoast

进入 slave 容器修改配置

$ docker exec -it keepalived_master bash # 进入 master 容器
$ vi /usr/share/nginx/html/index.html # 任意修改一点内容与 slave 做区分
$ nginx
$ curl localhoast

测试 Keepalived

启动 master 容器中的 keepalived

$ docker exec -it keepalived_master bash
$ keepalived
$ curl 172.17.0.200 # curl 172.17.0.210
$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
36: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.17.0.200/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.17.0.210/32 scope global eth0
       valid_lft forever preferred_lft forever

启动 slave 容器中的 keepalived

$ docker exec -it keepalived_slave bash
$ vi /etc/keepalived/keepalived.conf # state 改为 BACKUP priority 改为 小于 100 的任意值 比如 99
$ keepalived
$ curl 172.17.0.200 # curl 172.17.0.210

总结：

你会发现访问 172.17.0.200 以及 172.17.0.210 都是访问到 maser 容器上的内容
如果关闭 master 上的 keepalived，就会自动切换访问到 slave 容器上的内容
当 nginx 进程不存在时尝试重新启动 nginx，如果 nginx 启动失败就关闭 keepalived 服务并实现自动切换

遗留问题：

Docker 中设置的虚拟 IP，如何让外网访问，如何做映射呢？Docker 研究得不深入，请多多指教。
check.sh 脚本检查 Nginx 服务时有时间返回 0 有时候返回 2 (Nginx 服务都是关闭的状态)，但使用 touch 新建一个文件又没有问题！！！这是为何呢？

欢迎一起交流…

酷玩时刻博客专家

发布了98 篇原创文章 · 获赞 162 · 访问量 55万+

他的留言板关注