Service debug commands

As the traffic increases, many scf services will have problems. The following troubleshooting methods can be used as a reference. Although they are all very simple commands, they can be quickly located when the service has problems (non-scf services can also be used for reference)

1. View the system load cpu mem io load average ... These information can be viewed through the top, iostat, ifstat, jstat, xxstat... commands

II . View service process status

1. Process cpu occupancy:

    sort by cpu occupancy:

ps Hh -eo pid,tid,pcpu |sort -nk3|tail

    locate which thread occupies cpu

top -> shift + h List the thread list (see here The arrived tid is in decimal)
shift + t Sort by time
jstack pid (process number) > output to file
and then find the thread number in hexadecimal

    jstack command

jstack 12345 > jstack_dump

    pstree View all threads of the process

pstree 12345

2. mem usage

    top command to

    check the memory occupied by the java process.

Through the output of jmap, you can check whether there is a memory "leak" problem, which instance occupies more resources. jmap can also dump the memory of the entire process

jmap –histo 12345 > mem_dump

    Check the gc status:

jstat –gcutil pid time interval (see the attachment for details)

jstat –gcutil 12345 1000


3. File open status (everything under linux is a file)

    lsof

example: lsof –p pid |wc –l View process opened The number of files, this command can check whether there are resources in the program that are not closed, such as: socket file .... httpclient commonly used in java services is easy to forget to close, through this command, you can quickly locate

lsof -p 12345 |wc -l


4. View the network status

    netstat (see the attachment for details, this command is invincible)

Example: View the number of connections on port 16003:

netstat –na|grep 16003|wc –l

List all connections of each ip connected to 16003

netstat -an |grep 16003|awk '{print $5}'|awk -F : '{print $4}'|sort|uniq -c

output:

1 *

2 10.3.12.15
4 10.3.12.20
3 192.168.10.22
3 192.168.10.23


3 192.168.10.24

    tcpdump #Show

only the TCP sections with source and destination port 80
tcpdump 'tcp and port 80' #Only

display the source and destination port 80, and the TCP section with the SYN flag set
tcpdump 'tcp and port 80 and tcp[13:1] & 2 != 0'

#Only display the source port For the TCP section between 7001~7005
tcpdump 'tcp and tcp[0:2]>7000 and tcp[0:2]<=7005'

    telnet #Capture

http packets
telnet 10.58.120.118 80 > http_dump
GET /test/n_16170701962244 .jpg HTTP/1.1
Host: pic.58.com
Connection: Keep-Alive

#scf status monitoring

#interactive
telnet 10.58.120.110 26003 > 26003_dump
count

#pipeline
(echo -e "count";sleep 10)|telnet 10.58. 120.110 26003 #Network

communication through fd of linux
exec 6<>/dev/tcp/10.58.120.110/26003 #Connect fd=6 associated with 10.58.120.110:26003
echo -e "count">& 6 #write count
cat<&6 #View the received result
exec 6>&- #Close the input and output stream
exec 6<&-