Java threads: interpreting thread states of a running JVM

Question

A Java thread is always in one of the following ten states:

NEW: Just starting up, i.e., in process of being initialized.
NEW_TRANS: Corresponding transition state (not used, included for completness).
IN_NATIVE: Running in native code.
IN_NATIVE_TRANS: Corresponding transition state.
IN_VM: Running in VM.
IN_VM_TRANS: Corresponding transition state.
IN_JAVA: Running in Java or in stub code.
IN_JAVA_TRANS: Corresponding transition state (not used, included for completness).
BLOCKED: Blocked in vm.
BLOCKED_TRANS: Corresponding transition state.

The unused state (UNINITIALIZED) has been omitted from the list.

While the definitions of the states are given above I'm looking for "rule-of-thumbs" for interpreting a given thread state setup for a running appserver. And, more specifically:

Assume a live application server with the following thread statistics (obtained using jstack) at various points in time:

• 100 threads: 35 BLOCKED, 65 IN_NATIVE
• 113 threads: 35 BLOCKED, 77 IN_NATIVE, 1 IN_VM
• 52 threads: 38 BLOCKED, 1 IN_JAVA, 6 IN_NATIVE, 7 IN_VM
• 120 threads: 39 BLOCKED, 1 IN_JAVA, 80 IN_NATIVE
• 94 threads: 34 BLOCKED, 59 IN_NATIVE, 1 IN_NATIVE_TRANS

For each thread of the five statistics - what can be inferred with regards to the overall JVM state? I.e. "in this scenario the JVM looks to be idling waiting for requests", "the machine is busy processing requests", etc.

Answer 1

This level of output doesn't provide enough information to make such statements.

As an example, consider the BLOCKED state: there are many things that can cause a thread to be blocked. Two of them are waiting for data to come from a client, and waiting for data to come back from a database. In the first case, your application is idle, in the second it's overloaded.

Edit: not having looked at the output from jstack, I suppose that these two conditions could also be represented as IN_NATIVE. However, the same comment holds: you don't know what they're doing, so you can't make any statements about the application as a whole.

Answer 2

kdgregory is correct that thread state won't necessarily reveal what you want by itself. However, jstack should also give you stack traces, letting you see exactly where the threads are in your program (assuming it's not obfuscated or something). A BLOCKED thread whose trace contains a call to InputStream.read(), for example, should be fairly obvious.

Answer 3

I'd say what's interesting in general when looking at thread states or indeed profiling data in general is to be able to ask yourself "Did I expect this to be the case?" If you have no opinion on whether the data you're getting is bad/good/expected/unexpected, then it's difficult to do much about it.

With thread states, I think this is more interesting to look at the behaviour of individual threads and then ask yourself "did I expect that thread to be in that state/waiting for that lock for quite so long?" And just knowing that a given thread is blocked/waiting etc per se isn't so interesting as knowing WHAT it was blocked/waiting for.