netstat monitors a large number of ESTABLISHED connections and Time_Wait connection issues

Problem description:

Netstat monitors a large number of ESTABLISHED connections and Time_Wait connections without considering system load, CPU, memory, etc.

# netstat -n | awk '/^tcp/ {++y[$NF]} END {for(w in y) print w, y[w]}'
CLOSE_WAIT 348
ESTABLISHED 1240
TIME_WAIT 5621

Monitor the connection between Apache and tomcat Link port

#netstat -n | grep 8009 | wc -l

7198

Question 1: How to solve a lot of Time_Wait

by adjusting kernel parameters:
copy the code

vim /etc/sysctl.conf #Edit the
file and add the following:
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30 #Then
execute /sbin/sysctl -p to make the parameters take effect.

Copy code

Configuration instructions:

net.ipv4.tcp_syncookies = 1 means enable SYN Cookies. When the SYN waiting queue overflows, enable cookies to prevent a small number of SYN attacks. The default value is 0, which means close;

net.ipv4.tcp_tw_reuse = 1 means enabling reuse. Allow TIME-WAIT sockets to be reused for new TCP connections, the default is 0, which means close;

net.ipv4.tcp_tw_recycle = 1 means to enable fast recycling of TIME-WAIT sockets in TCP connections, the default is 0, which means close;

net. ipv4.tcp_fin_timeout=30 Modify the default TIMEOUT time of the system.

If the performance of the above configuration is not satisfactory after tuning, you can continue to modify the configuration:
copy the code

vi /etc/sysctl.conf
net.ipv4.tcp_keepalive_time = 1200
#Indicates the frequency of TCP sending keepalive messages when keepalive is enabled. The default is 2 hours, change to 20 minutes.
net.ipv4.ip_local_port_range = 1024 65000 #Indicates
the port range for outgoing connections. Small by default: 32768 to 61000, change to 1024 to 65000.
net.ipv4.tcp_max_syn_backlog = 8192 #Indicates
the length of the SYN queue, the default is 1024, and the increased queue length is 8192, which can accommodate more network connections waiting to be connected.
net.ipv4.tcp_max_tw_buckets = 5000
#Indicates that the system keeps the maximum number of TIME_WAIT sockets at the same time. If this number is exceeded, the TIME_WAIT socket will be cleared immediately and a warning message will be printed.
The default is 180000, change to 5000. For Apache, Nginx and other servers, the parameters in the above lines can reduce the number of TIME_WAIT sockets well, but for Squid, the effect is not great. This parameter can control the maximum number of TIME_WAIT sockets to prevent the Squid server from being dragged down by a large number of TIME_WAIT sockets.

Copy the code After the

tuning is complete, press it again to see the effect.

# netstat -n | awk '/^tcp/ {++y[$NF]} END {for(w in y) print w, y[w]}'

ESTABLISHED 968

Question 1: How to solve the problem that there are still a lot of ESTABLISHED has not been released The

preliminary inference is that there is a problem when the tomcat server recycles the session, which is generally related to the Timeout setting of the server.

View tomcat's configuration file server.xml



<Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" URIEncoding="UTF-8" />
*****



Check the configuration and find that acceptCount=”100” when it takes 20000 milliseconds, which is obviously unreasonable, and the maximum number of connections is too small.

So further optimization:

connectionTimeout="20000" is changed to connectionTimeout="100"

acceptCount="100" is changed to acceptCount="5000" The

optimization is completed, and the pressure test is continued... The

system responsiveness is rising, and LoadRunner reported an error until it overwhelmed the maximum concurrency. Never appeared again.

Action.c(380): Error -26608: HTTP Status Code=504 (Gateway Time-out) for "http://www.cnlogs.com/javame"

Summary:

TBD, write later!