jstack: Target process not responding

Question

I am running Ubuntu server edition and I wanted to take a thread dump of Tomcat.

So, I first tried to find out which PID tomcat uses:

$ jps -l
5809 sun.tools.jps.Jps

But it's not there?

So, I used top instead and found out the PID 5730.

Then I called jstack to get the thread dump:

$ sudo jstack -l 5730
5730: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

What's going on? :-(

I already tried to export CATALINA_TMPDIR as described in Jstack and Jstat stopped working with upgrade to JDK6u23 but that didn't change anything:

$ export CATALINA_TMPDIR=/tmp
$ sudo /etc/init.d/tomcat6 restart
 * Stopping Tomcat servlet engine tomcat6
   ...done.
 * Starting Tomcat servlet engine tomcat6
   ...done.
$ sudo jstack -l 5934 // new PID after restart
5934: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Update:

I also tried sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt but it only gives me tons of exceptions on the console.

Answer 1

I got it working by doing two things:

1. Changed call to: sudo -u tomcat6 jstack -J-d64 -m pid
2. Replaced OpenJDK with Sun's original sun-6-jdk and sun-6-jre packages

---

Explanation for part 1: I switched to 64-bit mode, used sudo and run the command as Tomcat user.

Note: Part 2 might not be necessary. For some users it seems like part 1 is enough. In fact, try to add just the sudo command first. It might already do the trick.

Answer 2

I think you need to run jstack as the same user that runs the Tomcat process. Note also that jps only returns processes for the current user. You would get the pid for the Tomcat process by running jps with sudo or as the Tomcat process user.

This bug report may also be useful: https://bugs.launchpad.net/ubuntu/+source/sun-java6/+bug/597098

Answer 3

@Valmar, I find the same topic post here. Unable to get thread dump? Any ideas why my app blocks?

It seems the workaround is sudo -u tomcat6 kill -3 <pid>.

Answer 4

This also worked for me:

sudo -u tomcat6 kill -3 pid

It looks like nothing happens but when you look in the logs the stacks are there. They look like exceptions if your not expecting them.

Answer 5

I find it useful to use something like 'ps -eo pid,user,command | grep java' to find the actual java command being used, then use the directory to find the matching jstack etc.

# ps -eo user,command | grep '[j]ava' | cut -d' ' -f1
someuser /usr/lib/jvm/java/bin/java

# /usr/lib/jvm/java/bin/java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

So its 64-bit, running as 'someuser'. su to that user and run run jstack etc. from that same directory. (ie. /usr/lib/jvm/java/bin/jstack

Useful when you're on a server with various different installations / implementations of Java.

Answer 6

Try to switch to process user and then use jstack:

sudo -u {process user} jstack > dump

Answer 7

I had same problem, but none of below solution worked for me:

jstack <pid>
jstack -J-d64 -m <pid>
sudo -u <user> jstack ...

I finally upgraded JDK from jdk1.6.0_24 to jdk1.7.0_67 and every things worked.

Answer 8

For Tomcat users having this issue, check your catalina.out log file.

I was having the same problems "22693: Unable to open socket file: target process not responding or HotSpot VM not loaded". I gave up and was trying to find anything about what happened before it locked up, but then there was the jstack output in the log file.

Answer 9

For those finding this answer six years after it was asked and they are on CentOS 7 note that SELinux will stop jmap from writing the heap dumps as, out of the box, it will deny writing to a socket unless the directory has the type tomcat_tmp_t.

In my case, I write my heap dumps under /usr/share/tomcat/.jmap. This directory is owned by my runtime user.

Therefore to allow jmap to write to this directory:

semanage fcontext -a -t tomcat_tmp_t "/usr/share/tomcat/.jmap(/.*)?"
restorecon /usr/share/tomcat -vR

Which allows us to then run, as our tomcat user:

jmap -dump:format=b,file=/usr/share/tomcat/.jmap/tomcat-`date +%s`.bin <pid>