Nginx 搭建LVS-DR模式

1 环境准备

1.1 服务器与IP规划

LVS服务器
VIP（虚拟IP）：192.168.51.103
DIP（转发者IP/内网IP）：192.168.51.4

Nginx服务器两台（RealServer）
RIP（真实IP/内网IP）：192.168.51.5
RIP（真实IP/内网IP）：192.168.51.6

1.2 停止NetworkManager服务

三台服务器分别执行以下两行命令，因为服务器使用的是虚拟机，需要停止 NetworkManager，不然会出现问题

[root@localhost network-scripts]# systemctl stop NetworkManager
[root@localhost network-scripts]# systemctl disable NetworkManager

2 配置LVS节点与ipvsadm

2.1 创建子接口

LVS服务器

[root@localhost network-scripts]# cd /etc/sysconfig/network-scripts
[root@localhost network-scripts]# ls
ifcfg-ens33      ifcfg-lo     ifdown-eth   ifdown-isdn  ifdown-routes  ifdown-TeamPort  ifup-aliases  ifup-ippp  ifup-plip   ifup-ppp     ifup-Team      ifup-wireless      network-functions-ipv6
ifdown       ifdown-ippp  ifdown-post  ifdown-sit     ifdown-tunnel    ifup-bnep     ifup-ipv6  ifup-plusb  ifup-routes  ifup-TeamPort  init.ipv6-global
ifcfg-ens33.bak  ifdown-bnep  ifdown-ipv6  ifdown-ppp   ifdown-Team    ifup             ifup-eth      ifup-isdn  ifup-post   ifup-sit     ifup-tunnel    network-functions
[root@localhost network-scripts]# cp ifcfg-ens33 ifcfg-ens33:1

2.2 修改配置文件

[root@localhost network-scripts]# vi ifcfg-ens33:1

BOOTPROTO=static
DEVICE=ens33:1
ONBOOT=yes
IPADDR=192.168.51.103
NETMASK=255.255.255.0

2.3 刷新配置

[root@localhost network-scripts]# service network restart

2.4 安装ipvsadm

好像阿里云不支持虚拟IP,购买官网的负载均衡
好像腾讯云支持最大虚拟IP数量为10

ipvsadm -Ln

[root@localhost network-scripts]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

3 为两台RS配置虚拟IP

两台RS机器的配置相同，下面的操作需要在两台机器上都进行操作。

3.1 复制配置

[root@localhost network-scripts]# cd /etc/sysconfig/network-scripts
[root@localhost network-scripts]# ls
ifcfg-ens33       ifdown-eth   ifdown-isdn  ifdown-routes  ifdown-TeamPort  ifup-aliases  ifup-ippp  ifup-plip   ifup-ppp     ifup-Team      ifup-wireless      network-functions-ipv6
ifcfg-ens33.bak  ifdown       ifdown-ippp  ifdown-post  ifdown-sit     ifdown-tunnel    ifup-bnep     ifup-ipv6  ifup-plusb  ifup-routes  ifup-TeamPort  init.ipv6-global
ifcfg-lo         ifdown-bnep  ifdown-ipv6  ifdown-ppp   ifdown-Team    ifup             ifup-eth      ifup-isdn  ifup-post   ifup-sit     ifup-tunnel    network-functions
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:1

3.2 修改配置

[root@localhost network-scripts]# vi ifcfg-lo:1

DEVICE=lo:1
IPADDR=192.168.51.103
NETMASK=255.255.255.255
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback

3.3 刷新配置

方式一

[root@localhost network-scripts]# ifup lo

方式二

[root@localhost network-scripts]# service network restart

4 为两台RS配置arp

4.1 arp介绍

ARP响应级别与通告行为。

1.arp-ignore：ARP响应级别（处理请求）

0：只要本机配置了ip，就能响应请求
1：请求的目标地址到达对应的网络接口，才会响应请求

2.arp-announce：ARP通告行为（返回响应）

0：本机上任何网络接口都向外通告，所有的网卡都能接受到通告
1：尽可能避免本网卡与不匹配的目标进行通告
2：只在本网卡通告

4.2 修改配置文件

[root@localhost network-scripts]# vi /etc/sysctl.conf

# configration for lvs
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2

4.3 刷新配置

[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2

4.4 增加网关

[root@localhost network-scripts]# route add -host 192.168.51.103 dev lo:1

设置开机添加

[root@localhost network-scripts]# echo "route add -host 192.168.51.103 dev lo:1" >> /etc/rc.local

5 使用ipvsadm配置集群规则

5.1 创建LVS节点

用户访问数据的集群调度者

[root@localhost network-scripts]# ipvsadm -A -t 192.168.51.103:80 -s rr -p 5

-A：添加集群
-t：tcp协议
ip地址：设定集群的访问ip，也就是LVS的虚拟ip
-s：设置负载均衡的算法，rr表示轮询
-p：设置连接持久化的时间

5.2 创建2台RS真实服务器

[root@localhost network-scripts]# ipvsadm -a -t 192.168.51.103:80 -r 192.168.51.5:80 -g
[root@localhost network-scripts]# ipvsadm -a -t 192.168.51.103:80 -r 192.168.51.6:80 -g

5.3 保存到规则库

不保存，重启会失效

[root@localhost network-scripts]# ipvsadm -S

5.4 检查集群

查看集群列表

[root@localhost network-scripts]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.51.103:80 rr persistent 5
  -> 192.168.51.5:80              Route   1      0          0         
  -> 192.168.51.6:80              Route   1      0          0

查看集群状态

[root@localhost network-scripts]# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  192.168.51.103:80                   0        0        0        0        0
  -> 192.168.51.5:80                     0        0        0        0        0
  -> 192.168.51.6:80                     0        0        0        0        0

5.5 其他命令

# 重启ipvsadm，重启后需要重新配置
service ipvsadm restart
# 查看持久化连接
ipvsadm -Ln --persistent-conn
# 查看连接请求过期时间以及请求源ip和目标ip
ipvsadm -Lnc
# 设置tcp tcpfin udp 的过期时间（一般保持默认）
ipvsadm --set 1 1 1
# 查看过期时间
ipvsadm -Ln --timeout

详细的帮助文档

ipvsadm -h
man ipvsadm

6 相关信息

博文不易，辛苦各位猿友点个关注和赞，感谢