Fastjson with generic serialization leads to memory leaks
1. Background
One morning, a large number of slow interfaces with a response time of more than 1s appeared in the production environment alarm group, and many interfaces were found to be timed out and blown out in the application logs (response time>=5s).
Among them, the SQL of many interfaces can use indexes. After checking by DBA, there is no problem with the database.
Use the top command on the host to check the CPU usage, and find that there is an exception: the CPU of some hosts is around 50%, and the CPU of one of the hosts occasionally goes up to 300% and continues.
2. High CPU investigation
1. Check the PID of the process with the highest CPU consumption 41594
Shell command: top -c
2. Find out the thread number 41600 that consumes the most CPU according to the Pid
Shell command: top -Hp 41594
3. This is the decimal data, converted to hexadecimal as a280
Shell command: printf "%x\n" 41600
4. Export the stack information of PID
Shell command: jstack -l 41594 > 41594.stack
5. Search the thread number keyword "a280" in 41594.stack and find the following line
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f41f8026800 nid=0xa280 runnable
I searched for the top 5 thread numbers and found that they are all GC threads, indicating that there is a problem with the heap memory .
3. Monitoring situation
Check the JVM memory and GC status through Grafana monitoring, and find that the heap memory is basically full
GC times and time significantly increased
In this case, it is necessary to obtain a memory snapshot for analysis and export the memory snapshot
shell命令：jmap -dump:format=b,file=dump-20210809.hprof 41594
Download after compressing the file
shell命令：tar -czf dump-20210809.hprof.tar.gz dump-20210809.hprof
Shell command: sz dump-20210809.hprof.tar.gz
Three, dump analysis
Using the tool MemoryAnalyzer (MAT) to analyze the dump file, it is found that 1.4G of memory is occupied by the class com.alibaba.fastjson.parser.ParserConfig
Expand the list_objects view to view and find that 1.4G is consumed in com.alibaba.fastjson.util.IdentityHashMap
Query relevant information on the Internet, and found that there are also related problems: Improper use of fastjson deserialization leads to memory leaks - liqipeng - 博客园
Search for the keyword ' ParameterizedTypeImpl ' in the code , it is indeed useful
So far, the problem has been located, exactly the same as the information said.
The next step is to reproduce this problem, write a loop to execute this code continuously, loop 100,000 times, and then dump the heap memory to see.
The results are as follows, 346M are occupied by the class com.alibaba.fastjson.parser.ParserConfig
The performance of com.alibaba.fastjson.util.IdentityHashMap is also consistent
Online information also provides related solutions, here we use 'method 2'
Then dump the heap memory snapshot, com.alibaba.fastjson.parser.ParserConfig no longer appears in the view
