(2003, "Can't connect to MySQL server on '127.0.0.1' (99)")

• Cause: Large concurrent mysqlconnections, time_waitaccumulation leads to port exhaustion.
• View by
  
  • Use the following commands to count the number of connections in various current states
    
    • $ netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
      TIME_WAIT 1788
      ESTABLISHED 3
  • Possible scenarios and explanations
    
    • CLOSED: No connection is active or in progress
    • LISTEN: The server is waiting for an incoming call
    • SYN_RECV: A connection request has arrived, waiting for confirmation
    • SYN_SENT: the application has started, opening a connection
    • ESTABLISHED: Normal data transfer status
    • FIN_WAIT1: the app says it's done
    • FIN_WAIT2: The other side has agreed to release
    • ITMED_WAIT: wait for all groups to die
    • CLOSING: Both sides try to close at the same time
    • TIME_WAIT: The other side has initiated a release
    • LAST_ACK: wait for all groups to die

• Cause of time_wait
  When a tcp endpoint closes the tcp connection, a small control block will be maintained in memory to record the ip address and port number of the recently closed link. This information will only be maintained for a short period of time, usually the Twice the estimated maximum segment usage period (to become 2MSL, usually 2 minutes), during which time two connections with the same ip address and port number cannot be recreated.

• Solution

  • Reduce time_wait connection waiting time (solved by modifying kernel parameters)

    • Mainly modify two parameters

      tcp_tw_recycle - BOOLEAN
      Enable fast recycling TIME-WAIT sockets. Default value is 0.  
      It should not be changed without advice/request of technical  
      experts.  
      net.ipv4.tcp_tw_recycle=1就是打开快速 TIME-WAIT sockets 回收，即快速回收处于TIME-WAIT的连接，默认值是0，即关闭状态
      
      tcp_tw_reuse - BOOLEAN  
      Allow to reuse TIME-WAIT sockets for new connections when it is  
      safe from protocol viewpoint. Default value is 0.  
      It should not be changed without advice/request of technical  
      experts. 
      允许重新应用处于TIME-WAIT状态的socket用于新的TCP连接，默认值是0，即关闭状态
      
      net.ipv4.tcp_fin_timeout
      设置连接超时时间，单位是秒，默认值是60秒

    • Modification method
      $ vi /etc/sysctl.conf
      Write or modify the following content:
      net.ipv4.tcp_fin_timeout = 2
      net.ipv4.tcp_tw_recycle = 1
      net.ipv4.tcp_tw_reuse = 1

  • Delay (100ms)

    I’m not sure this is something you’ll see in Real Life (tm). Try running
    an overnight test to see whether sleeping for 100 milliseconds between
    connections makes the problem go away. If it does, then you are just
    running our of available TCP ports.

    There’s a delay period after a TCP connection is closed and before the
    same port number can be re-used by another local process.

    If you run your test as it is currently written and after it fails run

    netstat -an

    you should see a large number of connections in the TIME_WAIT state. If
    so then you probably have nothing much to worry about.
    regards
    Steve

  • Increase the number of client load generation machines, or this ensures that the client and server are using several virtual ip addresses in a loop to increase more connection combinations. In this scenario, because it is the main interaction with the database, you can create a database connection pool by creating a pool of connections. Multiplexing of myslq connections reduces the generation of new connections.