tags: java, troubleshooting, monitor,jvm
Summarized in one sentence: the original jdk that comes with command-line tool so easy to use, this article will detail.
1 Introduction
Monitoring of java applications, the most convenient is the direct use of ready-made tools jdk provided in the bin directory of the installed jdk, has provided a variety of command-line monitoring tools to monitor and diagnose problems on java application developers and operation and maintenance personnel, Thus, such tools are an important means of java application monitoring. But also as java developers need to master basic skills.
2 common monitoring command-line tool
In general, common command-line tools include jps
, jinfo
, jmap
, jstack
, jstat
, these tools are in JAVA_HOME/bin/
the directory, the following outline:
jps
View java process IDjinfo
Check and adjust the parameters of the virtual machinejmap
View Heap (heap) and the use of raw heap snapshotjstack
View thread running state and generates a snapshot of the threadjstat
Show the process of class loading, operating data memory, garbage collection and so on.
With these tools, you can basically understand the change in the state memory java application thread running status and other information, and then provide the basis for application monitoring and problem diagnosis. The following examples will combine the use of these tools will be explained in detail, the sample code used herein is java-monitor-example
uploaded to my GitHub , address: https://github.com/mianshenglee
.
3 Process Query Tooljps
3.1 jps
Description
To monitor java application, the first step is to know which application process, what its operating parameters Yes. jps
It is that you can process the query tool. Linux familiar with the students, probably all know the query process uses ps -ef|grep java
such a command, jps is similar, but it does not use name lookup, but to find all java jdk processes that are currently running, and only find the current user of the Java process, rather than the current system in all processes.
3.2 jps
Use
As a command-line tool, you can -help
view the parameter Help is also available on the official document : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jps.html
as follows:
[root@test bin]# jps -help
usage: jps [-help]
jps [-q] [-mlvV] [<hostid>]
Definitions:
<hostid>: <hostname>[:<port>]
参数解释:
-q:只显示java进程的pid
-m:输出传递给main方法的参数,在嵌入式jvm上可能是null
-l:输出应用程序main class的完整package名 或者 应用程序的jar文件完整路径名
-v:输出传递给JVM的参数
复制代码
Examples of projects java-monitor-example
up and running in a linux machine, use jps
can output the following information:
- Only the process ID
[root@test bin]# jps -q
13680
14214
复制代码
- The full name of the program output and JVM parameters
[root@test bin]# jps -lv
13680 java-monitor-example-0.0.1-SNAPSHOT.jar -Xms128m -Xmx128m -Dserver.port=8083
14289 sun.tools.jps.Jps -Denv.class.path=.:/opt/jdk8/lib:/opt/jdk8/jre/lib -Dapplication.home=/opt/jdk8
复制代码
The contents of the output in java-monitor-example-0.0.1-SNAPSHOT.jar
is -l
the full name of output, -Xms128m -Xmx128m -Dserver.port=8083
is passed to the JVM parameters.
- Use the command in a shell script and get java process ID as a variable use
JAVA_HOME="/opt/jdk8"
APP_MAINCLASS=java-monitor-example
#初始化psid变量(全局)
psid=0
#查看进程ID函数
checkpid() {
javaps=`$JAVA_HOME/bin/jps -l | grep $APP_MAINCLASS`
if [ -n "$javaps" ]; then
psid=`echo $javaps | awk '{print $1}'`
else
psid=0
fi
}
#调用函数后通过psid进行业务逻辑操作,如根据进程id杀进程
checkpid
echo "(pid=$psid)"
复制代码
The above script, more suitable for operation and maintenance personnel to open and close applications, automatically get java process ID, and whether to run (start) based on the ID determination program, or close the application ( kill -9
).
4 configuration tooljinfo
4.1 jinfo
Description
Know the process ID belongs java application is the first step in the article "java application monitoring (2) -java secret command", the startup parameters already know there are a lot of java, java before monitoring applications need to understand clearly its What startup parameters Yes. Then you need to use jinfo
tools. jinfo
JVM may output system parameters and application parameters JAVA. jinfo also be able to modify the adjustable parameters during part of running virtual machines, many operating parameters can not be adjusted, if "can not be changed" abnormal, indicating that can not be adjusted. But the official documentation states that this command may no longer be used in subsequent releases, current JDK8 can still be used.
4.2 jinfo
Use
By -help
viewing parameters help is also available on jinfo
the official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jinfo.html
as follows:
[root@test bin]# jinfo -help
Usage:
jinfo [option] <pid>
(to connect to running process)
where <option> is one of:
-flag <name> to print the value of the named VM flag
-flag [+|-]<name> to enable or disable the named VM flag
-flag <name>=<value> to set the named VM flag to the given value
-flags to print VM flags
-sysprops to print Java system properties
<no option> to print both of the above
-h | -help to print this help message
复制代码
Use jps
After obtaining the application process ID (PID example is 13,680), directly if jps <pid>
all of the system parameters and JVM is output parameters, other parameters described help
also said very clearly. Or a combination of the following sample code java-monitor-example
to practice what:
- The initial value of the acquisition heap java applications
[root@test bin]# jinfo -flag InitialHeapSize 13680
-XX:InitialHeapSize=134217728
复制代码
- See all the JVM arguments
[root@test bin]# jinfo -flags 13680
Attaching to process ID 13680, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.51-b03
Non-default VM flags: -XX:CICompilerCount=2 -XX:InitialHeapSize=134217728 -XX:MaxHeapSize=134217728 -XX:MaxNewSize=44564480 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=44564480 -XX:OldSize=89653248 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseParallelGC
Command line: -Xms128m -Xmx128m -Dserver.port=8083
复制代码
Visible, because we set the start -Xms
and -Xmx
that is the corresponding -XX:InitialHeapSize
and -XX:MaxHeapSize
value. In addition, the parameters -Dserver.port
are system parameters, jinfo -sysprops 13680
you can view the system parameters.
5 heap viewerjmap
5.1 jmap
Description
After java application starts, it runs in the JVM, memory is where the need to focus on monitoring, jmap
is one such tool that takes a snapshot of a running jvm heap, including the overall situation, the heap occupancy histograms, dump out snapshot files for offline analysis. Official documentation states that this command may no longer be used in subsequent releases, current JDK8 can still be used.
5.2 jmap
Use
By -help
viewing parameters help is also available on jmap
the official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jmap.html
help as follows:
[root@test bin]# jmap -help
Usage:
jmap [option] <pid>
(to connect to running process)
where <option> is one of:
<none> to print same info as Solaris pmap
-heap to print java heap summary
-histo[:live] to print histogram of java object heap; if the "live"
suboption is specified, only count live objects
-clstats to print class loader statistics
-finalizerinfo to print information on objects awaiting finalization
-dump:<dump-options> to dump java heap in hprof binary format
dump-options:
live dump only live objects; if not specified,
all objects in the heap are dumped.
format=b binary format
file=<file> dump heap to <file>
Example: jmap -dump:live,format=b,file=heap.bin <pid>
-F force. Use with -dump:<dump-options> <pid> or -histo
to force a heap dump or histogram when <pid> does not
respond. The "live" suboption is not supported
in this mode.
-h | -help to print this help message
-J<flag> to pass <flag> directly to the runtime system
复制代码
As indicated above, jmap
parameters are used -heap
, -histo
and -dump
, in conjunction with examples java-monitor-example
, as follows:
- Print jvm memory usage overall
[root@test bin]# jmap -heap 13680
Attaching to process ID 13680, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.51-b03
using thread-local object allocation.
Parallel GC with 2 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 134217728 (128.0MB)
NewSize = 44564480 (42.5MB)
MaxNewSize = 44564480 (42.5MB)
OldSize = 89653248 (85.5MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 31981568 (30.5MB)
used = 5306632 (5.060798645019531MB)
free = 26674936 (25.43920135498047MB)
16.59278244268699% used
From Space:
capacity = 6291456 (6.0MB)
used = 1081440 (1.031341552734375MB)
free = 5210016 (4.968658447265625MB)
17.18902587890625% used
To Space:
capacity = 6291456 (6.0MB)
used = 0 (0.0MB)
free = 6291456 (6.0MB)
0.0% used
PS Old Generation
capacity = 89653248 (85.5MB)
used = 16615680 (15.845947265625MB)
free = 73037568 (69.654052734375MB)
18.533271655701753% used
18006 interned Strings occupying 2328928 bytes.
复制代码
From the above information, it can be seen the JVM heap memory currently used, including the young generation ( Eden
area, From
region, To
area), and the old generation.
- View class name, number of objects, the size of the object occupy histogram
[root@test bin]# jmap -histo:live 13680|more
num #instances #bytes class name
----------------------------------------------
1: 36536 6462912 [C
2: 35557 853368 java.lang.String
3: 7456 826968 java.lang.Class
4: 20105 643360 java.util.concurrent.ConcurrentHashMap$Node
5: 1449 469024 [B
6: 6951 399280 [Ljava.lang.Object;
7: 9311 297952 java.util.HashMap$Node
8: 3122 274736 java.lang.reflect.Method
9: 2884 269112 [I
10: 6448 257920 java.util.LinkedHashMap$Entry
11: 2994 255160 [Ljava.util.HashMap$Node;
12: 15249 243984 java.lang.Object
.....
.....
复制代码
As described above, using -histo
output including the serial number, number of instances, the number of bytes occupied by the class name. Specified as follows:
- instances column: Indicates how many instances of the current class.
- bytes columns: Description of the current instance of the class total number of occupied bytes
- class name column: is the name of the current class representation, class name read:
- B represents a byte
- C represents char
- D means double
- F means float
- I on behalf of int
- J represents the long
- Z represents boolean
- [Representative array, such as [I corresponds to the int []
- Object with the [L + represents the class name
- The memory situations dump memory to a local file
[root@test bin]# jmap -dump:file=./heap.hprof 13680
复制代码
As described above, the stack will be written to the current directory where the heap.hprof
file as to how to analyze this document, may be used jhat
, but typically the actual development, rarely jhat memory dump file to be analyzed directly, so it is no longer They are told. More is the use of tools MAT
to visualize way to view, follow-up article will be on MAT
the use of tools will be explained in detail.
6 thread stack query tooljstack
6.1 jstack
Description
This command prints the specified Java application thread stack for each Java frame, will print the full class names, method names, byte code index (BCI) and line number, can be used to detect deadlocks, thread standstill, the process consumes investigation cpu too high alarm issues.
6.2 jstack
Use
Use -help
parameter access help, is also available on jstack
the official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstack.html
help as follows:
[root@test bin]# jstack -help
Usage:
jstack [-l] <pid>
(to connect to running process)
jstack -F [-m] [-l] <pid>
Options:
-F 强制dump线程堆栈信息. 用于进程hung住, jstack <pid>命令没有响应的情况
-m 同时打印java和本地(native)线程栈信息,m是mixed mode的简写
-l 打印锁的额外信息
复制代码
Binding example java-monitor-example
, the thread information can be printed (usually prints the contents of the file and then written to the analysis), as follows:
- Print this thread stack information
[root@test bin]# jstack 13680
2019-08-16 23:18:18
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.51-b03 mixed mode):
"http-nio-8083-Acceptor-0" #39 daemon prio=5 os_prio=0 tid=0x00007f7520698800 nid=0x359a runnable [0x00007f7508bb7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000000f8c85380> (a java.lang.Object)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:745)
复制代码
6.3 thread dump analysis
6.3.1 Thread State
java thread stack using jstack
the dump out, you can see the state of the thread, the thread state was divided into six kinds, you can refer to the official documentation : https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr034.html
The following
- NEW
It has created a new thread, but has not started (not yet started), jstack
it will not print thread information of this state.
- RUNNABLE
Java virtual machine threads are run under the task of the state, but in fact it is just means that the thread is runnable (ready). For single-core CPU, multiple threads at the same time, can only run one thread, others need to wait for CPU scheduling.
- BLOCKED
The thread is blocked, waiting for a lock, a plurality of threads sharing a lock, a thread is used when the synchronization method of the lock to enter a synchronized block or method, and this thread needs to enter the sync block, the need to lock, cause the thread is blocked.
- WAITING
Wait state, the thread is in a wait state when executing the method of any of the three methods. 1. Object.wait
A method, use and no timeout parameter; 2 Thread.join
method is not used 3. timeout parameter LockSupport.park
method. Thread is waiting state will wait for another thread to handle special behavior. A thread is in a wait state (the wait, typically in waiting for other threads to complete an operation (or notifyAll Notify). Note that, Object.wait()
the method can only be called a synchronization code block. Called wait()
the method will release the lock.
- TIMED_WAITING
Thread waits the specified time, for calling this method may lead to a thread in this state:. 1 Thread.sleep 2. The method of Object.wait
method, with a time 3. The Thread.join
method 4. With time LockSupport.parkNanos
method, with time 5. LockSupport.parkUntil
Method with time. Note that Thread.sleep
the method is called, it will not release the lock, still occupy system resources.
- TERMINATED
Thread suspended state, this thread has fully carried out its task.
This can be seen from the following changes in thread state:
6.3.2 Analysis of jstack
the thread stack content
From the front using the jstack
dump out the information, we need to know the following information:
"http-nio-8083-Acceptor-0" #39
: Is the thread's name, therefore, we need to set their own general identification can name when creating a thread.daemon
Does it mean that thread is a daemon threadprio
Indicates the priority we set for the threados_prio
The priority of the corresponding operating system threads represented because not all operating systems support thread priorities, it may occur are set to 0tid
Thread idnid
Corresponding to the thread operating system native thread id, java each thread has a corresponding thread of the operating system, which is 16 hexadecimal, and therefore usually after the thread ID is acquired in the operating system, converted to hexadecimal need to correspond on.java.lang.Thread.State: RUNNABLE
Running, it has been described above state of the thread, if WAITING state, in brackets shows the causes of waiting, such as parking instructions because LockSupport.park method results in a call waiting. The usual stack information, there will be a lock mark, as- locked <0x00000000f8c85380> (a java.lang.Object)
said it is occupied by the lock.- For thread pause, CPU usage and other issues, the focus can look at the
wait
thread state - For the deadlock in the Dump out the thread stack snapshots can report directly to the Java level deadlock.
7 JVM Statistics tooljstat
7.1 jstat
Description
jstat
Namely JVM Statistics Monitoring Tool
, that JVM statistics monitoring tools, including monitoring class loading, memory, garbage collection, JIT compiler and other operating data, no graphics on the server, it is the tool of choice to run the positioning of virtual machine performance problems.
7.2 jstat
Use
Use -help
parameter access help, is also available on jstat
the official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html
help as follows:
[root@test bin]# jstat -help
jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]
Usage: jstat -help|-options
jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]
Definitions:
<option> An option reported by the -options option
<vmid> Virtual Machine Identifier. A vmid takes the following form:
<lvmid>[@<hostname>[:<port>]]
Where <lvmid> is the local vm identifier for the target
Java virtual machine, typically a process id; <hostname> is
the name of the host running the target Java virtual machine;
and <port> is the port number for the rmiregistry on the
target host. See the jvmstat documentation for a more complete
description of the Virtual Machine Identifier.
<lines> Number of samples between header lines.
<interval> Sampling interval. The following forms are allowed:
<n>["ms"|"s"]
Where <n> is an integer and the suffix specifies the units as
milliseconds("ms") or seconds("s"). The default units are "ms".
<count> Number of samples to take before terminating.
-J<flag> Pass <flag> directly to the runtime system.
复制代码
As shown in the above, vmid
, interval
, count
are the process ID, the print time interval (s or MS), the number of prints, wherein the option
main parameters are the following (the command may be used jstat -option
to view):
-class
Statistical class loader behavior information, such as total number of classes loaded-compile
Statistics HotSpot Just-in-Time compiler behavior-gc
Statistical information jdk when gc heap-gccapacity
Statistics different respective heap capacity situation generations-gccause
Gc statistics, and (with -gcutil) and cause gc events-gcnew
When statistics gc, the situation of the new generation-gcnewcapacity
When statistics gc, the new generation heap capacity-gcold
When statistics gc, situation of older areas-gcoldcapacity
When statistics gc, tenured heap capacity-gcpermcapacity
When statistics gc, permanent heap area capacity-gcutil
When statistics gc, heap situation-printcompilation
hotspot compiling statistical method
Generally, we use the -class
, -gc
, -gccause
and -gcutil
more, it is mainly used to analyze classes and heap usage and gc situation.
GC 7.3 The monitoring of the JVM
The example above works java-monitor-example
, for example, which contains a function to test memory overflow (using an array loop to create objects until memory overflow). Use jstat -gc 13680 1000
that is monitored every second, call /monitor/user/oom
the interface, the GC heap and see the changes. For the convenience of viewing, I put the output into sublime display, as shown below:
OOM error log output:
Output the contents of the above described each column as follows:
S0C
The current generation of young first survivor (s0) of total capacity (KB).S1C
A first current younger generation survivor (s1) of the total capacity (KB).S0U
s0 capacity (KB) has been used.S1U
s1 capacity (KB) has been used.EC
The current generation of young eden Chief capacity (KB).EU
eden area capacity (KB) has been used.OC
Total capacity Capacity (KB) older generations.OU
The old generation used capacity (KB).MC
Current Metaspace total capacity (KB).MU
Metaspace currently used capacity (KB).CCSC
Compressed class size capacityCCSU
Compressed class used capacityYGC
From when the application starts up to now, the total number of the young generation of young generation GC Events of occurrence.YGCT
From when the application starts up to now, a total elapsed time the young generation Young generation garbage collection.FGC
From when the application starts up to now, full GC events total number.FGCT
When the application starts from now, Full sc .GCT total time from when the application starts up to now, the total garbage collection timeGCT
.GCT = YGCT + FGCT
As can be seen from the above output line 6, EC
and EU
, OC
and OU
represents the young generation, old generation memory have been exhausted (and capacity values are equal), the OOM occurs. At this time, it is necessary to take measures to increase the memory (-Xmx parameter) or find the cause of the OOM code changes.
8 summary
For the monitoring of java applications, the paper jdk provides a command-line tool itself provides is explained and introduced the use of a complete description of the view java application process, view the boot parameters, view memory condition, see the thread, the view memory statistics, etc. mainly jps
, , jinfo
, jmap
, jstack
fivejstat
tools, and with examples, want to learn java developers to be able to master these techniques in the monitoring of java application, you can calmly face as OOM, high CPU, thread pause and other issues.
Reference material
- JDK Tools Reference document :
https://docs.oracle.com/javase/8/docs/technotes/tools/unix/
- Sample code
java-monitor-example
:https://github.com/mianshenglee/my-example/tree/master/java-monitor-example
- Java developers must master online issue of the investigation command :
https://www.hollischuang.com/archives/1561
- Java comes with performance monitoring tools :
http://www.tianshouzhi.com/api/tutorials/jvm/346