By analyzing a problem I'm trying to understand this strange stack trace:
Thread 3049: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.io.PipedInputStream.read() @bci=142, line=326 (Compiled frame)
- java.io.PipedInputStream.read(byte[], int, int) @bci=43, line=377 (Compiled frame)
- org.apache.http.entity.InputStreamEntity.writeTo(java.io.OutputStream) @bci=75, line=140 (Compiled frame)
- org.apache.http.impl.execchain.RequestEntityProxy.writeTo(java.io.OutputStream) @bci=10, line=123 (Compiled frame)
- org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(org.apache.http.HttpEntityEnclosingRequest) @bci=31, line=156 (Compiled frame)
- org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(org.apache.http.HttpEntityEnclosingRequest) @bci=5, line=162 (Compiled frame)
- org.apache.http.protocol.HttpRequestExecutor.doSendRequest(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=223, line=238 (Compiled frame)
- org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest, org.apache.http.HttpClientConnection, org.apache.http.protocol.HttpContext) @bci=25, line=123 (Compiled frame)
- org.apache.http.impl.execchain.MainClientExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) @bci=714, line=271 (Compiled frame)
- org.apache.http.impl.execchain.ProtocolExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) @bci=447, line=184 (Compiled frame)
- org.apache.http.impl.execchain.RetryExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) @bci=39, line=88 (Compiled frame)
- org.apache.http.impl.client.InternalHttpClient.doExecute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=168, line=184 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.doReleaseShared() @bci=69, line=695 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(int) @bci=9, line=1342 (Compiled frame)
- java.util.concurrent.CountDownLatch.countDown() @bci=5, line=291 (Compiled frame)
- com.mycompany.browse.concurrent.stream.CustomPipedInputStream.close() @bci=8, line=43 (Compiled frame)
- com.mycompany.browse.mina.handler.BrowseHttpMessageProcessor$1.run() @bci=57, line=200 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=745 (Compiled frame)
Based on the stack trace, AbstractQueuedSynchronizer calls Apache HTTP client. How could this happen? We are running Oracle Java 1.8 on Amazon Linux
And I have a lot of threads with exactly the same stacktrace. Anytime.
Later Edit: renamed from impossible to Inconceivable as suggested
As far as I see, you've got this stack trace in forced mode: jstack -F. There is a big difference between jstack and jstack -F described here.
The forced mode works out-of-process, asynchronously and without the cooperation from JVM. It is less reliable than regular jstack, because JVM is not at safepoint and its state may be inconsistent. For example, it can be in the middle of garbage collection.
The second reason is that stack dump is reconstructed by the external tool (Serviceability Agent) which is not quite perfect. So, in order to get reliable stack traces, I suggest to use jstack in normal mode when possible.