一、LVS+Keepalived 高可用群集
在这个高度信息化的 IT 时代，企业的生产系统、业务运营、销售和支持，以及日常管理等环节越来越依赖于计算机信息和服务，对高可用（HA）技术的应用需求不断提高，以便提供持续的、不间断的计算机系统或网络服务。
Keepalived是一个基于VRRP协议来实现的LVS服务高可用方案，可以解决静态路由出现的单点故障问题。
1、Keepalived工具介绍
专为LVS和HA设计的一款健康检查工具
• 支持故障自动切换（Failover）
• 支持节点健康状态检查（Health Checking）
• 官方网站：http://www.keepalived.org/
2、工作原理
在一个LVS服务集群中通常有主服务器(MASTER)和备份服务器(BACKUP)两种角色的服务器，但是对外表现为一个虚拟IP，主服务器会发送VRRP通告信息给备份服务器，当备份服务器收不到VRRP消息的时候，即主服务器异常的时候，备份服务器就会接管虚拟IP，继续提供服务，从而保证了高可用性。
3、Keepalived实现原理剖析
Keepalived采用VRRP热备份协议实现Linux服务器的多机热备功能
4、 VRRP （虚拟路由冗余协议）
是针对路由器的一种备份解决方案
由多台路由器组成一个热备组，通过共用的虚拟IP地址对外提供服务
每个热备组内同时只有一台主路由器提供服务，其他路由器处于冗余状态
若当前在线的路由器失效，则其他路由器会根据设置的优先级自动接替虚拟IP地址，继续提供服务
5、故障转移机制
Keepalived高可用服务之间的故障切换转移，是通过VRRP 来实现的。
在 Keepalived服务正常工作时，主 Master节点会不断地向备节点发送（多播的方式）心跳消息，用以告诉备Backup节点自己还活着，当主 Master节点发生故障时，就无法发送心跳消息，备节点也就因此无法继续检测到来自主 Master节点的心跳了，于是调用自身的接管程序，接管主Master节点的 IP资源及服务。
而当主 Master节点恢复时，备Backup节点又会释放主节点故障时自身接管的IP资源及服务，恢复到原来的备用角色。
二、LVS+Keepalived 高可用群集的搭建步骤
环境准备
主DR 服务器： 192.168.229.90
备DR 服务器： 192.168.229.80
Web 服务器1： 192.168.229.70
Web 服务器2： 192.168.229.60
nfs 服务器： 192.168.229.50
客户端：192.168.229.200
vip：192.168.229.100
1、配置负载调度器（主、备相同；192.168.229.90；192.168.229.80）
systemctl stop firewalld.service
setenforce 0
yum -y install ipvsadm keepalived
modprobe ip_vs
cat /proc/net/ip_vs
1.1 配置keeplived（主、备DR 服务器上都要设置）
cd /etc/keepalived/
cp keepalived.conf keepalived.conf.bak
vim keepalived.conf
......
global_defs { #定义全局参数
--10行--修改，邮件服务指向本地
smtp_server 127.0.0.1
--12行--修改，指定服务器（路由器）的名称，主备服务器名称须不同，主为LVS_01，备为LVS_02
router_id LVS_01
}
vrrp_instance VI_1 { #定义VRRP热备实例参数
--20行--修改，指定热备状态，主为MASTER，备为BACKUP
state MASTER
--21行--修改，指定承载vip地址的物理接口
interface ens33
--22行--修改，指定虚拟路由器的ID号，每个热备组保持一致
virtual_router_id 10
--23行--修改，指定优先级，数值越大优先级越高，主为100，备为99
priority 100
advert_int 1 #通告间隔秒数(心跳频率)
authentication { #定义认证信息,每个热备组保持一致
auth_type PASS #认证类型
--27行--修改，指定验证密码，主备服务器保持一致
auth_pass abc123
}
virtual_ipaddress { #指定群集vip地址
192.168.229.100
}
}
--36行--修改，指定虚拟服务器地址(VIP)、端口，定义虚拟服务器和Web服务器池参数
virtual_server 192.168.229.100 80 {
delay_loop 6 #健康检查的间隔时间(秒)
lb_algo rr #指定调度算法，轮询(rr)
--39行--修改，指定群集工作模式，直接路由(DR)
lb_kind DR
persistence_timeout 50 #连接保持时间(秒)
protocol TCP #应用服务采用的是 TCP协议
--43行--修改，指定第一个Web节点的地址、端口
real_server 192.168.229.90 80 {
weight 1 #节点的权重
--45行--删除，添加以下健康检查方式
TCP_CHECK {
connect_port 80 #添加检查的目标端口
connect_timeout 3 #添加连接超时(秒)
nb_get_retry 3 #添加重试次数
delay_before_retry 4 #添加重试间隔
}
}
real_server 192.168.229.80 80 { #添加第二个 Web节点的地址、端口
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 4
}
}
##删除后面多余的配置##
}
systemctl start keepalived
ip addr show dev ens33 #查看虚拟网卡vip
1.2 配置分发策略（#keepalived配置好后，可以不用再配ipvsadm 分发策略）
ipvsadm-save > /etc/sysconfig/ipvsadm
systemctl start ipvsadm
ipvsadm -C
ipvsadm -A -t 192.168.229.100:80 -s rr
ipvsadm -a -t 192.168.229.100:80 -r 192.168.229.70:80 -g
ipvsadm -a -t 192.168.229.100:80 -r 192.168.229.60:80 -g
ipvsadm
ipvsadm -ln
ipvsadm-save > /etc/sysconfig/ipvsadm
#如果没有vip的分发策略。则重启keepalived 服务
1.3 调整内核 proc 响应参数，关闭linux内核的重定向参数响应
vim /etc/sysctl.conf
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
sysctl -p
2、部署共享存储（NFS服务器：192.168.229.50）
systemctl stop firewalld.service
systemctl disable firewalld.service
setenforce 0
yum install nfs-utils rpcbind -y
systemctl start nfs.service
systemctl start rpcbind.service
systemctl enable nfs.service
systemctl enable rpcbind.service
mkdir /opt/accp /opt/benet
chmod 777 /opt/accp /opt/benet
vim /etc/exports
/usr/share *(ro,sync)
/opt/accp 192.168.229.0/24(rw,sync)
/opt/benet 192.168.229.0/24(rw,sync)
--发布共享---
exportfs -rv
3、配置节点服务器（192.168.229.70；192.168.229.60）
systemctl stop firewalld
setenforce 0
yum -y install httpd
systemctl start httpd
vim /etc/sysconfig/network-scripts/ifcfg-lo:0
DEVICE=lo:0
ONBOOT=yes
IPADDR=192.168.229.100
NETMASK=255.255.255.255
service network restart 或 systemctl restart network
ifup lo:0
ifconfig lo:0
route add -host 192.168.229.100 dev lo:0
vim /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
sysctl -p
--192.168.229.70---
mount.nfs 192.168.229.50:/opt/accp /var/www/html
--192.168.229.60---
mount.nfs 192.168.229.50:/opt/benet /var/www/html
4、测试验证
在客户端访问 http://192.168.229.100/ ，默认网关指向 192.168.229.100
再在主服务器关闭 keepalived 服务后在测试,systemctl stop keepalived
三、实例操作：LVS+Keepalived 高可用群集
1、配置负载调度器（主、备相同；192.168.229.90；192.168.229.80）
1.1 关闭防火墙并安装ipvsadm和keepalived工具
主DR 服务器： 192.168.229.90
[root@192 ~]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install ipvsadm keepalived
[root@192 ~]# modprobe ip_vs
[root@192 ~]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
备DR 服务器： 192.168.229.80
[root@192 ~]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install ipvsadm keepalived
[root@192 ~]# modprobe ip_vs
[root@192 ~]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
1.2 配置keeplived（主、备DR 服务器上都要设置）
主DR 服务器： 192.168.229.90
[root@192 ~]# cd /etc/keepalived/
[root@192 keepalived]# ls
keepalived.conf
[root@192 keepalived]# cp keepalived.conf{,.bak}
[root@192 keepalived]# ls
keepalived.conf keepalived.conf.bak
[root@192 keepalived]# vim keepalived.conf
[root@192 keepalived]# systemctl start keepalived.service
[root@192 keepalived]# ip addr show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:4a:3e:ba brd ff:ff:ff:ff:ff:ff
inet 192.168.229.90/24 brd 192.168.229.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.229.100/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::d20e:a6c8:5f4a:ce2b/64 scope link
valid_lft forever preferred_lft forever
备DR 服务器： 192.168.229.80
[root@192 ~]# cd /etc/keepalived/
[root@192 keepalived]# ls
keepalived.conf
[root@192 keepalived]# cp keepalived.conf{,.bak}
[root@192 keepalived]# ls
keepalived.conf keepalived.conf.bak
[root@192 keepalived]# vim keepalived.conf
[root@192 keepalived]# systemctl start keepalived.service
[root@192 keepalived]# ip addr show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:4a:3e:ba brd ff:ff:ff:ff:ff:ff
inet 192.168.229.90/24 brd 192.168.229.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.229.100/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::d20e:a6c8:5f4a:ce2b/64 scope link
valid_lft forever preferred_lft forever
1.3 调整内核 proc 响应参数，关闭linux内核的重定向参数响应
主DR 服务器： 192.168.229.90
[root@192 keepalived]# vim /etc/sysctl.conf
[root@192 keepalived]# sysctl -p
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
备DR 服务器： 192.168.229.80
[root@192 keepalived]# vim /etc/sysctl.conf
[root@192 keepalived]# sysctl -p
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
2、部署共享存储（NFS服务器：192.168.229.50）
[root@192 ~]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# rpm -q rpcbind nfs-utils
rpcbind-0.2.0-42.el7.x86_64
nfs-utils-1.3.0-0.48.el7.x86_64
[root@192 ~]# mkdir /opt/ly /opt/weq
[root@192 ~]# echo '<h1>This is node server1!</h1>' > /opt/ly/index.html
[root@192 ~]# echo '<h1>This is node server2!</h1>' > /opt/weq/index.html
[root@192 ~]# ls /opt/ly /opt/weq
/opt/ly:
index.html
/opt/weq:
index.html
[root@192 ~]# vim /etc/exports
[root@192 ~]# chmod 777 /opt/ly /opt/weq
[root@192 ~]# showmount
clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)
[root@192 ~]# showmount -e
clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)
[root@192 ~]# systemctl start rpcbind
[root@192 ~]# systemctl start nfs
[root@192 ~]# systemctl enable rpcbind
[root@192 ~]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@192 ~]# exportfs -rv
exporting 192.168.229.0/24:/opt/weq
exporting 192.168.229.0/24:/opt/ly
[root@192 ~]# showmount -e
Export list for 192.168.229.50:
/opt/weq 192.168.229.0/24
/opt/ly 192.168.229.0/24
3、配置节点服务器（192.168.229.70；192.168.229.60）
Web 服务器1： 192.168.229.70
[root@192 ~]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install httpd
[root@192 ~]# systemctl start httpd.service
[root@192 ~]# cd /etc/sysconfig/network-scripts/
[root@192 network-scripts]# ls
ifcfg-ens33 ifdown-ib ifdown-ppp ifdown-tunnel ifup-ib ifup-plusb ifup-Team network-functions
ifcfg-lo ifdown-ippp ifdown-routes ifup ifup-ippp ifup-post ifup-TeamPort network-functions-ipv6
ifdown ifdown-ipv6 ifdown-sit ifup-aliases ifup-ipv6 ifup-ppp ifup-tunnel
ifdown-bnep ifdown-isdn ifdown-Team ifup-bnep ifup-isdn ifup-routes ifup-wireless
ifdown-eth ifdown-post ifdown-TeamPort ifup-eth ifup-plip ifup-sit init.ipv6-global
[root@192 network-scripts]# vim ifcfg-lo
[root@192 network-scripts]# cat ifcfg-lo
DEVICE=lo:0
IPADDR=192.168.229.100
NETMASK=255.255.255.255
ONBOOT=yes
[root@192 network-scripts]# ifup lo:0
[root@192 network-scripts]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.229.100 netmask 255.255.255.255
loop txqueuelen 1 (Local Loopback)
[root@192 network-scripts]# route add -host 192.168.229.100 dev lo:0
[root@192 network-scripts]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.229.2 0.0.0.0 UG 100 0 0 ens33
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
192.168.229.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.229.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
[root@192 network-scripts]# vim /etc/sysctl.conf
[root@192 network-scripts]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@192 network-scripts]# showmount -e 192.168.229.50
Export list for 192.168.229.50:
/opt/weq 192.168.229.0/24
/opt/ly 192.168.229.0/24
[root@192 network-scripts]# mount.nfs 192.168.229.50:/opt/ly /var/www/html/
[root@192 network-scripts]# ls /var/www/html/
index.html
[root@192 network-scripts]# cat /var/www/html/index.html
<h1>This is node server1!</h1>
Web 服务器2： 192.168.229.60
[root@192 ~]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install httpd
[root@192 ~]# systemctl start httpd.service
[root@192 ~]# cd /etc/sysconfig/network-scripts/
[root@192 network-scripts]# ls
ifcfg-ens33 ifdown-ib ifdown-ppp ifdown-tunnel ifup-ib ifup-plusb ifup-Team network-functions
ifcfg-lo ifdown-ippp ifdown-routes ifup ifup-ippp ifup-post ifup-TeamPort network-functions-ipv6
ifdown ifdown-ipv6 ifdown-sit ifup-aliases ifup-ipv6 ifup-ppp ifup-tunnel
ifdown-bnep ifdown-isdn ifdown-Team ifup-bnep ifup-isdn ifup-routes ifup-wireless
ifdown-eth ifdown-post ifdown-TeamPort ifup-eth ifup-plip ifup-sit init.ipv6-global
[root@192 network-scripts]# vim ifcfg-lo
[root@192 network-scripts]# cat ifcfg-lo
DEVICE=lo:0
IPADDR=192.168.229.100
NETMASK=255.255.255.255
ONBOOT=yes
[root@192 network-scripts]# ifup lo:0
[root@192 network-scripts]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.229.100 netmask 255.255.255.255
loop txqueuelen 1 (Local Loopback)
[root@192 network-scripts]# route add -host 192.168.229.100 dev lo:0
[root@192 network-scripts]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.229.2 0.0.0.0 UG 100 0 0 ens33
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
192.168.229.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.229.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
[root@192 network-scripts]# vim /etc/sysctl.conf
[root@192 network-scripts]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@cm network-scripts]# showmount -e 192.168.229.50
Export list for 192.168.229.50:
/opt/weq 192.168.229.0/24
/opt/ly 192.168.229.0/24
[root@cm network-scripts]# mount.nfs 192.168.229.50:/opt/weq /var/www/html/
[root@cm network-scripts]# ls /var/www/html/
index.html
[root@cm network-scripts]# cat /var/www/html/index.html
<h1>This is node server2!</h1>
4、测试验证
在客户端访问 http://192.168.229.100/ ，默认网关指向 192.168.229.100
再在主服务器关闭 keepalived 服务后在测试,systemctl stop keepalived
再在主服务器关闭 keepalived 服务后在测试,systemctl stop
