- Overview of JVM Monitoring and Diagnostics
Various problems are often encountered during program operation. When locating problems, it is usually necessary to synthesize various information, such as system logs, heap dump files, thread dump files, and GC logs. Through virtual machine monitoring and diagnosis tools, we can quickly obtain and analyze the data we need, thereby improving the speed of problem solving. You can use graphical tools such as jconsole, jvisualvm, or commands to monitor and diagnose commands. This article will introduce the use of common monitoring and problem diagnosis command tools for virtual machines, including the following tools:
| jps | Display all Hotspot virtual machine processes in the system |
| jstat | Collect all aspects of running data of the Hotspot virtual machine |
| jstack | Display the thread stack information of the virtual machine |
| jinfo | Display the configuration information of the virtual machine |
| jmap | Memory snapshot information for generating virtual machines |
- Command introduction
1). jps command
JVM Process Status Tool, this command is used to list running virtual machine processes, showing the name of the main class and the virtual machine process id. This command is affected by the access rights of the current user. For example, non-root users in Linux only list the virtual machine processes started by the current user.
Command format:
jps [options] [hostid]
Execution example:
glassfish@0220:~$ jps -l 32214 sun.tools.jps.Jps 11850 com.sun.enterprise.glassfish.bootstrap.ASMain
Common parameters:
| -l | output main class full name |
| -v | Output the jvm parameters of the virtual machine process startup |
| -m | Arguments passed to the main function when output starts |
2).jstack command
Stack Trace for Java is used to generate the current thread snapshot information of the virtual machine, including the stack information of each thread. This command is usually used to locate the cause of thread stall. When a thread stall occurs, you can view the stack information of each thread through stack to analyze the stall reason.
Command format:
jstack [ option ] pid
Common parameters:
| -l | Displays additional information about the lock in addition to the stack |
| -F | Force output thread stack when request is not answered |
| -m | Mixed mode, print stack information of java and native C++ calls |
Execution example:
"Thread-11" #24 daemon prio=5 os_prio=0 tid=0x0000000001b55800 nid=0x2e67 waiting on condition [0x00007ff666bfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e1b8ccb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Thread-9" #21 daemon prio=5 os_prio=0 tid=0x0000000001b0a000 nid=0x2e64 waiting on condition [0x00007ff66c346000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e1e1a310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.sun.common.util.logging.LoggingOutputStream.log(LoggingOutputStream.java:144)
at com.sun.common.util.logging.LoggingOutputStream$1.run(LoggingOutputStream.java:123)
"Thread-8" #20 daemon prio=5 os_prio=0 tid=0x0000000001b08800 nid=0x2e63 waiting on condition [0x00007ff66c447000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e1e0b4e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.sun.common.util.logging.LoggingOutputStream.log(LoggingOutputStream.java:144)
at com.sun.common.util.logging.LoggingOutputStream$1.run(LoggingOutputStream.java:123)
"pool-1-thread-1" #18 daemon prio=5 os_prio=0 tid=0x0000000001af4800 nid=0x2e61 waiting on condition [0x00007ff66c649000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e1a12528> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Timer-0" #13 daemon prio=5 os_prio=0 tid=0x00007ff6844f7000 nid=0x2e5c in Object.wait() [0x00007ff66cdf8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e15087b0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop (Timer.java medium26)
- locked <0x00000000e15087b0> (a java.util.TaskQueue)
at java.util.TimerThread.run (Timer.java:505)
"FelixDispatchQueue" #11 daemon prio=5 os_prio=0 tid=0x00007ff64400e800 nid=0x2e5a in Object.wait() [0x00007ff66e36a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063)
- locked <0x00000000e0c8da80> (a java.util.ArrayList)
at org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
at org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007ff6841ef000 nid=0x2e57 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ff6841d2000 nid=0x2e56 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ff6841d0000 nid=0x2e55 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ff684135800 nid=0x2e54 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ff684105000 nid=0x2e53 in Object.wait() [0x00007ff66eff6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x00000000e0c753b8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ff684103000 nid=0x2e52 in Object.wait() [0x00007ff66f0f7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000000e048ac30> (a java.lang.ref.Reference$Lock)
"VM Thread" os_prio=0 tid=0x00007ff6840fc000 nid=0x2e51 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff684023000 nid=0x2e4f runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ff684025000 nid=0x2e50 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007ff6841f1800 nid=0x2e58 waiting on condition
JNI global references: 298
3) jstat command
JVM Statistics Monitoring Tool, commands for monitoring various running status information. This tool is useful in environments that only have a text console, such as production environments in an enterprise. It can be used to display operating data such as class loading, garbage collection, and runtime compilation status in the system.
Command format:
jstat [ generalOption | outputOptions vmid [interval[s|ms] [count]] ]
vmid represents the unique identifier of the virtual machine. If it is a local virtual machine process, it is consistent with the LVMID, which is usually the local virtual machine process number. interval indicates the query interval, and count indicates the number of queries. If the interval and count parameters are omitted, the query is performed once.
Common parameters:
| class | Class loading related information. |
| compiler | Methods compiled by the JIT compiler, time-consuming, etc. |
| gc | Java heap information and garbage collection status. |
| gccapacity | Pay attention to the maximum and minimum space of each area of the java heap. |
| gccause | Similar to gcutil, extra output causes the last gc. |
| gcnew | Cenozoic GC status. |
| gcnewcapacity | Pay attention to the maximum and minimum space used by the young generation gc. |
| gcold | Old age gc status. |
| gcoldcapacity | Pay attention to the maximum and minimum space used by the old generation gc. |
| gcpermcapacity | Pay attention to the maximum and minimum space used by the persistent generation gc. |
| gcutil | Pay attention to the proportion of used space to total space. |
| printcompilation | Outputs methods that have been JIT compiled. |
Execution example
glassfish@0220:~$ jstat -gcutil 11850 1000 3 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 66.10 96.76 84.62 86.22 77.39 6404 119.294 158 103.270 222.564 0.00 66.10 96.76 84.62 86.22 77.39 6404 119.294 158 103.270 222.564 0.00 66.10 97.15 84.62 86.22 77.39 6404 119.294 158 103.270 222.564
S0 and S1 represent Survivor0 and Survivor1, E represents the new generation Eden, O represents the old generation Old, M represents the size of the metaspace, and the above parameter values represent the proportion of used space. ccs compresses the class space, YGC represents the times of young gc, and YGCT represents the total time of young gc. FGC represents the times of Full gc, and FGCT represents the total time of full gc. GCT represents the total time spent by all gcs. Before jdk8: P means persistent generation Permanent
4) jinfo command
Configuration Info for Java, used to view and modify various parameter information of the virtual machine.
Command format:
jinfo [ option ] pid
Common parameters:
| -flag name | Print the value corresponding to this parameter of the virtual machine. |
| -flag [+\-]name | Enable or disable this parameter. |
| -flag name=value | Modify the value of the corresponding parameter. |
| -flags | 打印传给jvm的参数值. |
| -sysprops | 打印System.getProperties()信息. |
5) jmap命令
Memory Map for Java，可以产生堆dump文件，查询堆和持久代的详细信息等。
命令格式:
jmap [ option ] pid
常用参数:
| -dump | 生成堆dump文件，格式为: -dump:[live,]format=b,file=<filename> |
| -heap | 显示java堆的详细信息，包括垃圾回收期、堆配置和分代信息等 |
| -histo | 显示堆中对象的统计信息，包括类名称，对应的实例数量和总容量 |
| -permstat | 统计持久代中各ClassLoader的统计信息。 |
示例：
jmap -dump:format=b,file=dump.tmp 11850 Dumping heap to /home/glassfish/dump.tmp ... Heap dump file created上面这个命令生成了dump.tmp这个dump文件。生成的dump文件可以使用Eclipse Memory Analyzer/jhat等工具进行分析
6) jhatk命令
Jvm Heap Analysis Tool， 与jmap一起使用
示例：
jhat dump.tmp Reading from dump.tmp... Dump file created Tue Jun 28 13:55:09 CST 2016
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.sun.tools.hat.internal.model.Snapshot.makeId(Snapshot.java:586)
at com.sun.tools.hat.internal.model.Snapshot.addHeapObject(Snapshot.java:166)
at com.sun.tools.hat.internal.parser.HprofReader.readInstance(HprofReader.java:737)
at com.sun.tools.hat.internal.parser.HprofReader.readHeapDump(HprofReader.java:484)
at com.sun.tools.hat.internal.parser.HprofReader.read(HprofReader.java:238)
at com.sun.tools.hat.internal.parser.Reader.readFile(Reader.java:92)
at com.sun.tools.hat.Main.main(Main.java:159)
- 遇到的问题
Attaching to process ID 3538, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
Solution:
1. echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
This method is valid until the next restart.
2. Permanently effective method
sudo vi /etc/sysctl.d/10-ptrace.conf
Edit the following line:
kernel.yama.ptrace_scope = 1
to:
kernel.yama.ptrace_scope = 0
Restart the system to make the modification take effect.
For details of this bug, please refer to the official documentation: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7050524
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.sun.tools.hat.internal.model.Snapshot.makeId(Snapshot.java:586)
at com.sun.tools.hat.internal.model.Snapshot.addHeapObject(Snapshot.java:166)
at com.sun.tools.hat.internal.parser.HprofReader.readInstance(HprofReader.java:737)
at com.sun.tools.hat.internal.parser.HprofReader.readHeapDump(HprofReader.java:484)
at com.sun.tools.hat.internal.parser.HprofReader.read(HprofReader.java:238)
at com.sun.tools.hat.internal.parser.Reader.readFile(Reader.java:92)
at com.sun.tools.hat.Main.main(Main.java:159)