As the number of visits increases, the default capacity of the system reaches the upper limit, and system logs appear frequently. The common "kernel: TCP: time wait bucket table overflow" message in /var/log/messages will show several lines every 5s. At this point, check the connection status as follows:
[root@IPTV001 ~]# netstat -an | awk '{print $6}' | sort | uniq -c | sort -rn
6216 TIME_WAIT
30 LISTEN
21 ESTABLISHED
20 CONNECTED
14
4 STREAM
2 FIN_WAIT1
1 I-Node
1 Foreign
1 FIN_WAIT2
1 established)
1 CLOSE_WAIT
1 and
1 987101641
1 8614
1 8613
1 8597
can see that the TIME_WAIT value is still relatively large
Modify vi /etc/sysctl.conf
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max=65535
fs.inotify.max_user_instances = 8192
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_max_tw_buckets = 20000
net.ipv4.route.gc_timeout = 100
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_max_orphans = 262144
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 1200
Note:
net.ipv4.tcp_syncookies = 1 means enable SYN Cookies. When the SYN waiting queue overflows, enable cookies to prevent a small number of SYN attacks. The default value is 0, which means close;
net.ipv4.tcp_tw_reuse = 1 means enabling reuse. Allow TIME-WAIT sockets to be reused for new TCP connections, the default is 0, which means close;
net.ipv4.tcp_tw_recycle = 1 means that fast recycling of TIME-WAIT sockets in TCP connections is enabled, and the default is 0, which means close.
net.ipv4.tcp_fin_timeout = 30 means that if the socket is requested to be closed by the local end, this parameter determines how long it will remain in the FIN-WAIT-2 state.
net.ipv4.tcp_keepalive_time = 1200 indicates how often TCP sends keepalive messages when keepalive is enabled. The default is 2 hours, change to 20 minutes.
net.ipv4.ip_local_port_range = 1024 65000 represents the port range used for outgoing connections. Small by default: 32768 to 61000, change to 1024 to 65000.
net.ipv4.tcp_max_syn_backlog = 65536 indicates the length of the SYN queue, the default is 1024, and the increased queue length is 65536, which can accommodate more network connections waiting to be connected.
net.ipv4.tcp_max_tw_buckets = 20000 means that the system keeps the maximum number of TIME_WAIT sockets at the same time. If this number is exceeded, the TIME_WAIT socket will be cleared immediately and a warning message will be printed. The default is 180000, change to 5000. For Apache, Nginx and other servers, the parameters in the above lines can reduce the number of TIME_WAIT sockets well, but for Squid, the effect is not great. This parameter can control the maximum number of TIME_WAIT sockets to prevent the Squid server from being dragged down by a large number of TIME_WAIT sockets.
Execute the following command to make the configuration take effect:
sysctl -p
After observing for a few minutes, the error disappeared and the number of TIME_WAIT connections decreased.
root@IPTV001 ~]# netstat -an | awk '{print $6}' | sort | uniq -c | sort -rn
4962 TIME_WAIT
30 LISTEN
22 ESTABLISHED
20 CONNECTED
14
4 STREAM
2 FIN_WAIT1
1 I-Node
1 Foreign
1 established)
1 and
1 987101641
1 8614
1 8613
1 8597