Online case analysis - the java process is normal but the request is blocked

1. Problem phenomenon

When the user opens the page, the screen goes blank. After 30 seconds, many interfaces report request timeouts.

Two, the cause of the problem

The database connection pool in the java process is exhausted. The specific reasons are high delay in calling the third-party interface and the database of the other party's application is down.

3. Investigation process

1. Check the connection data volume of each IP

SELECT id,client_ip,COUNT(client_ip) AS client_num FROM 
(SELECT SUBSTRING_INDEX(HOST,':' ,1) AS client_ip,id FROM information_schema.processlist ) AS connect_info 
GROUP BY client_ip ORDER BY client_num DESC;

2. Find all threads whose execution time exceeds 5 minutes and piece together the kill statement to facilitate subsequent inspection and killing.

SELECT CONCAT('kill ', id, ';') FROM information_schema.processlist WHERE Command != 'Sleep' AND TIME > 300 ORDER BY TIME DESC;

4. Solution

Just introduce a circuit breaker to the project