如上图所示，102和103是内网nginx服务器，100和101是边界LB，clinet是1，这个实验是为了实现在LB上虚拟出一个VIP，client通过访问该VIP，来动态负载到两台内网nginx服务器上面来。流量的来回，都需要经过边界LB。
我们首先分析一下，这种模式是一种NAT模式，流量的来回都要经过边界LB。回忆一下NAT的工作模式，LB在接收到client的数据请求后，根据自身负载均衡算法，将目标IP地址修改为内网RS地址，然后从内网口扔出去，内网服务器收到这个数据请求之后，目标MAC和IP都是自己，源地址是client，这个时候就会处理该数据包，返回，返回的数据包中，源地址是自己，目标地址是client，在经过LB的时候，会被转换，源地址为VIP地址，目标地址为client地址，然后client就会收到这个回包。
这个和物理硬件LB的工作模式中，透明模式相同，透明模式也就是NAT模式。而物理硬件中的反向代理模式，和LVS中的tunnel模式相似，在接收到client的数据请求后，会将整个数据包进行改写，源地址为LB的内网地址，目标地址为某一个RS地址，然后数据包返回的时候，也就会再次进行转换。这种反向代理模式中，服务器正常情况下无法知道client的源，只知道是自己的LB访问了自己，可以通过配置LB，添加字段，来讲该client的地址一并发送给RS。
物理硬件LB中的三角模式，也就是LVS的DR模式。需要在服务器上配置lo接口。
回到环境中。这个地方采用的是NAT模式，也就是说流量必须要经过LB，最简单的做法，就是将两台NGINX服务器的网关，配置在LB上，而且是以VIP的形式存在，对外，LB也拿出一个VIP，作为client访问的入口，也就是说，LB上，有两个VRRP INSTANCE，而且要保持状态一致。这样才能实现高可用HA。
下面开始做配置了，主要是在两台LB上配置keepalived，keepalived是默认安装的，配置文件存放在/etc/keepalived/目录下，启动脚本为/etc/init.d/keepalived，定义100为主，101为备。下面是两台设备的keepalived配置文件。
! Configuration File for keepalived ! The global defines,including notification server and target,router-id and others global_defs { notification_email { [email protected] 0633225522@domain.com } notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LB_100 } ! The vrrp sync group,any vrrp instance in the vrrp sync group while keep the same status, ! no matter the priority ,for example ,VI_1 is master ,VI_2 is backup,the sync group status is backup vrrp_sync_group VG1 { group { VI_To_client VI_To_server } } ! The VRRP instance for client to connect,clients connect to the VIPs vrrp_instance VI_To_client { state MASTER interface eth0 virtual_router_id 192 priority 150 nopreempt advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.245.200 } } ! The VRRP instance for real servers,real servers` default gateway vrrp_instance VI_To_server { state MASTER interface eth1 virtual_router_id 10 priority 150 nopreempt advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.254 } } ! The LB configuration,including VS and RS,scheduling argu,and other options,health check virtual_server 192.168.245.200 80 { delay_loop 6 lb_algo rr lb_kind NAT persistence_timeout 10 protocol TCP real_server 10.0.0.102 80 { weight 1 TCP_CHECK { connect_timeout 3 } } real_server 10.0.0.103 80 { weight 1 TCP_CHECK { connect_timeout 3 } } }
在100的配置文件中，主要是配置了一个vrrp_sync_group，里面有两个VRRP INSTANCE，分别是面向client和面向RS的，两个VRRP INSTANCE的vrrp-_id是不同的，但是其他属性值都是ixiangtong的。另外，在keepalived的LB配置中，有对VS和RS的配置，同时对RS使用了health check，检查方法很简单，就是TCP的connect，当能够访问到时，将该RS动态的加入到ipvs中，如果访问不到，就从ipvs中移除。这个现象可以通过查看日志看到。
slave的配置文件和master的配置文件大同小异，除了必要的字段不同，其他都是相同的。下面看看keepalived的日志信息。
Jul 15 13:57:01 CentOS_101 Keepalived[26012]: Starting Keepalived v1.2.13 (03/19,2015) Jul 15 13:57:01 CentOS_101 Keepalived[26013]: Starting Healthcheck child process, pid=26015 Jul 15 13:57:01 CentOS_101 Keepalived[26013]: Starting VRRP child process, pid=26016 Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Netlink reflector reports IP 192.168.245.101 added Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Netlink reflector reports IP 192.168.245.101 added Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Netlink reflector reports IP 10.0.0.101 added Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Netlink reflector reports IP fe80::20c:29ff:fe88:718 added Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Netlink reflector reports IP fe80::20c:29ff:fe88:722 added Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Registering Kernel netlink reflector Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Registering Kernel netlink command channel Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Opening file '/etc/keepalived/keepalived.conf'. Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Configuration is using : 15025 Bytes Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Using LinkWatch kernel netlink reflector... Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Netlink reflector reports IP 10.0.0.101 added Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Netlink reflector reports IP fe80::20c:29ff:fe88:718 added Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Netlink reflector reports IP fe80::20c:29ff:fe88:722 added Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Registering Kernel netlink reflector Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Registering Kernel netlink command channel Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Registering gratuitous ARP shared channel Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Opening file '/etc/keepalived/keepalived.conf'. Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Configuration is using : 69565 Bytes Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: Using LinkWatch kernel netlink reflector... Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Activating healthchecker for service [10.0.0.102]:80 Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Activating healthchecker for service [10.0.0.103]:80 Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: VRRP_Instance(VI_To_client) Entering BACKUP STATE Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: VRRP_Instance(VI_To_server) Entering BACKUP STATE Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Jul 15 13:57:01 CentOS_101 Keepalived_vrrp[26016]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(12,13)]
这个是启动日志，在启动的时候，就对两个RS就开启了健康检查。
Jul 15 13:57:19 CentOS_101 Keepalived_healthcheckers[26015]: TCP connection to [10.0.0.102]:80 failed !!! Jul 15 13:57:19 CentOS_101 Keepalived_healthcheckers[26015]: Removing service [10.0.0.102]:80 from VS [192.168.245.200]:80 Jul 15 13:57:49 CentOS_101 Keepalived_healthcheckers[26015]: TCP connection to [10.0.0.102]:80 success. Jul 15 13:57:49 CentOS_101 Keepalived_healthcheckers[26015]: Adding service [10.0.0.102]:80 to VS [192.168.245.200]:80 Jul 15 13:57:59 CentOS_101 Keepalived_healthcheckers[26015]: TCP connection to [10.0.0.103]:80 failed !!! Jul 15 13:57:59 CentOS_101 Keepalived_healthcheckers[26015]: Removing service [10.0.0.103]:80 from VS [192.168.245.200]:80 Jul 15 13:58:29 CentOS_101 Keepalived_healthcheckers[26015]: TCP connection to [10.0.0.103]:80 success. Jul 15 13:58:29 CentOS_101 Keepalived_healthcheckers[26015]: Adding service [10.0.0.103]:80 to VS [192.168.245.200]:80
这个是测试RS故障时候，keepalived的日志。日志写的很清楚，当主机故障时，就从VS中移除，同时可以看到ipvsadm的状态。
[root@CentOS_100 keepalived]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.245.200:80 rr persistent 10 -> 10.0.0.102:80 Masq 1 0 0 [root@CentOS_100 keepalived]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.245.200:80 rr persistent 10 -> 10.0.0.102:80 Masq 1 0 0 -> 10.0.0.103:80 Masq 1 0 3 [root@CentOS_100 keepalived]# ipvsadm -L -n
client在访问的时候，只需要访问200这个虚拟地址就好了。如果不想LB作为RS的网关，可以直接在RS上写一条路由即可。
在NAT模式中，需要在LB上开启转发，先查看sysctl -p查看是否开启了转发，如果没有开启，需要在/etc/sysctl.conf中修改net.ipv4.ip_forward为1，默认为0，如果为0，数据包在经过LB的时候，是不会被转发出去的。这个时候的LB，作为的是一个路由器，用linux作为路由器，是需要开启这个属性的。