AIX6.1上WAS服务器CPU占用高的处理步骤

1、问题现象

服务器环境是AIX6.1\WAS7.0，通过topas监控CPU经常出现100%的情况，而且负荷比较轻时，CPU占用率不能也下降。

2、处理步骤

1、通过topas监控可以看到当前占用CPU高的两个java进程，记录下进程号：6619258；

2、通过ps -mp 6619258 -o THREAD 以查找当前进程正在占用 CPU 的线程信息，把输出信息拷贝到文本文件中；

3、通过kill -3 6619258 输出ThreadDump线程执行堆栈快照信息，在当前目录中找到类似javacore.20120322.130602.6619258.0003.txt文件，拷贝到本地；

4、下面分析这两个文件，分析tid线程信息发现有下面标红的几个线程占用CPU比较高：

USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND

root 6619258 1 - A 314 60 108 * 242001 - -1 /was/IBM/WebSphere/AppServer/java/bin/java -

- - - 10616851 S 0 82 1 f1000f0a1000a240 8410400 - -1 -

- - - 11141271 S 0 82 1 f1000f0a1000aa40 8410404 - -1 -

- - - 20971659 S 0 82 1 f1000f0a10014040 8410404 - -1 -

- - - 21168201 R 55 118 0 - 400000 - -1 -

- - - 21299399 S 0 82 1 f1000f0a10014540 8410400 - -1 -

- - - 22544397 S 0 82 1 f1000f0a10015840 8410400 - -1 -

- - - 29294701 S 0 82 1 f1000f0a1001bf40 8410400 - -1 -

- - - 29819073 S 0 82 1 f1000f0a1001c740 8410400 - -1 -

- - - 29884629 R 72 130 0 - 400000 - -1 -

- - - 30212313 S 0 82 1 f1000f0a1001cd40 8410400 - -1 -

- - - 30277691 S 0 82 1 f1000f0a1001ce40 8410400 - -1 -

- - - 34013245 S 0 82 1 f1000f0a10020740 8410404 - -1 -

- - - 34078781 S 0 82 1 f1000f0a10020840 8410400 - -1 -

- - - 34209851 R 69 128 0 - 400000 - -1 -

- - - 34406643 S 1 82 1 f1000f0a10020d40 8410400 - -1 -

- - - 34537533 S 0 82 1 - 418400 - -1 -

5、取这些线程号，转换为16进制的数据，21168201 转换16进制值为0x1430049，可以使用windows的“计算器”工具，先输入10进制数值，然后切换到16进制。

取0x1430049在javacore.20120322.130602.6619258.0003.txt中找到下面一段日志：

3XMTHREADINFO "WebContainer : 3" TID:0x00000000E188B100, j9thread_t:0x000000011A5F0D20, state:CW, prio=5

3XMTHREADINFO1 (native thread ID:0x1430049, native priority:0x5, native policy:UNKNOWN)

4XESTACKTRACE at org/apache/regexp/RECompiler.setNextOfEnd(RECompiler.java:209(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RECompiler.expr(RECompiler.java:1193(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RECompiler.terminal(RECompiler.java:866(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RECompiler.closure(RECompiler.java:942(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RECompiler.branch(RECompiler.java:1151(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RECompiler.expr(RECompiler.java:1203(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RECompiler.compile(RECompiler.java:1281(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RE.<init>(RE.java:495(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RE.<init>(RE.java:480(Compiled Code))

4XESTACKTRACE at com/sbdcpn/utils/Stringvalidator.getRegularObj(Stringvalidator.java:173(Compiled Code))

4XESTACKTRACE at com/sbdcpn/utils/Stringvalidator.validValue(Stringvalidator.java:150(Compiled Code))

4XESTACKTRACE at com/sbdcpn/utils/tools/StringFormat.StringFormatToDate(StringFormat.java:40(Compiled Code))

4XESTACKTRACE at com/sbdcpn/page/utils/PageUtilMethod.getDataValue(Unknown Source:279(Compiled Code))

4XESTACKTRACE at com/sbdcpn/page/taglib/IteratorResultTag.convertFormat(IteratorResultTag.java:697(Compiled Code))

4XESTACKTRACE at com/sbdcpn/page/taglib/IteratorResultTag.formatResult(IteratorResultTag.java:671(Compiled Code))

4XESTACKTRACE at com/sbdcpn/page/taglib/IteratorResultTag.doAfterBody(IteratorResultTag.java:372(Compiled Code))

4XESTACKTRACE at com/ibm/_jsp/_grid_5F_plan_5F_query._jspService(_grid_5F_plan_5F_query.java:1889(Compiled Code))

29884629 转换为16进制值为0x1C800D5，查找日志为：

3XMTHREADINFO "WebContainer : 8" TID:0x00000000E22E7600, j9thread_t:0x000000011FBE1500, state:CW, prio=5

3XMTHREADINFO1 (native thread ID:0x1C800D5, native priority:0x5, native policy:UNKNOWN)

4XESTACKTRACE at org/apache/regexp/RECompiler.setNextOfEnd(RECompiler.java:207(Compiled Code))

4XESTACKTRACE at org/apache/regexp/RECompiler.expr(RECompiler.java:1194(Compiled Code))