Thread dump containing threads in RUNNABLE state with no stack

Question

Doing a thread dump on a highly loaded application with CPU, I see a lot of threads in this state:

"ajp-executor-threads - XXXXXX" prio=10 tid=0x00002b04b8b33801 nid=0x5327 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

What is really strange to me is that there is no stacktrace at all and that total number of ajp-thread is higher than max-threads (below) configured

It is happening with an application running on:

• JBoss 7
• Java 7u75
• Redhat 5.11
• Running on VMWare Enterprise / vSphere 5.5

Configuration of executor is:

 <subsystem xmlns="urn:jboss:domain:threads:1.1">
        <bounded-queue-thread-pool name="ajp-executor">
            <core-threads count="32"/>
            <queue-length count="1"/>
            <max-threads count="300"/>
            <keepalive-time time="5" unit="seconds"/>
        </bounded-queue-thread-pool>
    </subsystem>

Note that load is very high as on this host:

• CPU reaches 70%

• Load is at 4 (== number of vCPU)

Note these threads are not idle threads as an idle thread has this stack trace:

 "Reference Handler" daemon prio=5 tid=0x00007f92cb00e800 nid=0x3703 in Object.wait() [0x000000012057e000]
     java.lang.Thread.State: WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     - waiting on <0x00000007aaa84470> (a java.lang.ref.Reference$Lock)
     at java.lang.Object.wait(Object.java:503)
     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
     - locked <0x00000007aaa84470> (a java.lang.ref.Reference$Lock)

Answer 1

After further analysis I found that issue is due to remote debugging being enabled through:

-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=XXXXXXX

This explains these strange empty stack traces in thread dump.

Answer 2

Those are your idle threads in the AJP executor connection pool. You have set the core threads to 32 meaning that the pool will always maintain 32 threads in the connection pool, although they may be idle. With your configuration you could see up to 300 threads but any thread over 32 will only wait 5 seconds before dying and being removed from the connection pool.

As for you CPU load, I doubt its in any way related to these idle connection pool threads.

See John Skeet's answer here for some more info on connection pools and keep alive: https://stackoverflow.com/a/10379348/91866