jstack: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)

Question

We're using Wildfly 11 with Java 8 on Amazon Linux. We recently installed jstack to troubleshoot a high CPU utilization issue as we're trying to figure out what code is causing the CPU to spin. First we got the PID of the Wildfly process...

[myuser@prodmachine ~]$ ps -elf | grep java
0 S jboss     1992     1  0  80   0 - 28275 -      Aug30 ?        00:00:00 /bin/sh /usr/java/wildfly/bin/standalone.sh -c standalone.xml
0 S jboss     2044  1992 45  80   0 - 7336044 -    Aug30 ?        5-13:38:33 /usr/java/default/bin/java -D[Standalone] -server -Xms64m -Xmx25600m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=1024m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman,com.newrelic -java.awt.headless=true -javaagent:/usr/java/wildfly/newrelic/newrelic.jar -Dorg.jboss.boot.log.file=/usr/java/wildfly/standalone/log/server.log -Dlogging.configuration=file:/usr/java/wildfly/standalone/configuration/logging.properties -jar /usr/java/wildfly/jboss-modules.jar -mp /usr/java/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/usr/java/wildfly -Djboss.server.base.dir=/usr/java/wildfly/standalone -c standalone.xml
0 S 602       3630  1884  0  80   0 - 27617 pipe_w 14:19 pts/1    00:00:00 grep --color=auto java

However when I run the jstack command, I get this strange error:

[myuser@prodmachine ~]$ sudo /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-8.b13.39.39.amzn1.x86_64/bin/jstack -F 1992
[sudo] password for myuser: 
Attaching to process ID 1992, please wait...
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
    at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:411)
    at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
    at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
    at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
    at sun.tools.jstack.JStack.main(JStack.java:106)

I can't believe there's no SO post about this. Anyhow not sure what's going on and appreciate feedback.

Answer 1

First off I believe you were running the jstack with the wrong PID - it should be the one for the java process. So instead of PID 1992, you should try:

sudo jstack 2044

That said, I also ran into this issue, even when specifying the correct PID. The stack trace / heap dump failed when running the commands as the root user. Changing the current user to the one that owns the process worked!

Try running:

sudo su - jboss
jstack 2044

Answer 2

I have the same problem at the moment, but I have a different problem before I made sure jstack was a similar/matching version. That would be my suggestion. Right now, i'm using a jstack 1.8_191 but my JVM is 1.8_181, which I wouldn't think is an issue, but maybe that's the same as your issue. Hope that helps.

Answer 3

Thanks for this question, I came to this issue recently.

Doing some tests with some minor versions of OpenJDK 1.8, I was able to confirm that the issue is not on OpenJDK jstack, which has the -F as shown below, in Tests section . The issue is a SA difference on the JDK that you compiled the project and the one you're using to debug, on your case OpenJDK 1.8.181.

The workaround is to compile again the project with this new JDK you're using, in this case, OpenJDK.

Causes

After some research, I able to to find this book , Java Performance Comparison by Charlie Hunt et al., which describes this issue, or a very similar one, the description is below and the cause is that the Serviceability Agent, SA, is not able to open the transported core files.

When using SA when transported core files, we may get failures related to rtld_db or libthread_db mismatch_db mismatch, or SA may throw an error that some HotSpot symbol is missing on the target process. The stack provided is 

Stackframe

Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process)
        at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:411)
        at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
        at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
        at sun.tools.jstack.JStack.main(JStack.java:106)

TESTS Below

OPENJDK-1.8.0.191

[fdemeloj@fdemeloj jdk1.8.0_181]$ cd bin/
[fdemeloj@fdemeloj bin]$ ls
appletviewer  javafxpackager  jdb      jrunscript    pack200      unpack200
ControlPanel  javah           jdeps    jsadebugd     policytool   wsgen
extcheck      javap           jhat     jstack        rmic         wsimport
idlj          javapackager    jinfo    jstat         rmid         xjc
jar           java-rmi.cgi    jjs      jstatd        rmiregistry
jarsigner     javaws          jmap     jvisualvm     schemagen
java          jcmd            jmc      keytool       serialver
javac         jconsole        jmc.ini  native2ascii  servertool
javadoc       jcontrol        jps      orbd          tnameserv
[fdemeloj@fdemeloj bin]$ ./jstack --help
Usage:
    jstack [-l] <pid>
        (to connect to running process)
    jstack -F [-m] [-l] <pid>
        (to connect to a hung process)
    jstack [-m] [-l] <executable> <core>
        (to connect to a core file)
    jstack [-m] [-l] [server_id@]<remote server IP or hostname>
        (to connect to a remote debug server)

Options:
    -F  to force a thread dump. Use when jstack <pid> does not respond (process is hung)
    -m  to print both java and native frames (mixed mode)
    -l  long listing. Prints additional information about locks
    -h or -help to print this help message

OPENJDK-1.8.0.222

[fdemeloj@fdemeloj java-1.8.0-openjdk-1.8.0.222.b10-4.static.jdk.openjdkportable.x86_64]$ cd bin/
[fdemeloj@fdemeloj bin]$ ls
appletviewer  java          jconsole  jps         native2ascii  schemagen   xjc
clhsdb        javac         jdb       jrunscript  orbd          serialver
extcheck      javadoc       jdeps     jsadebugd   pack200       servertool
hsdb          javah         jhat      jstack      policytool    tnameserv
idlj          javap         jinfo     jstat       rmic          unpack200
jar           java-rmi.cgi  jjs       jstatd      rmid          wsgen
jarsigner     jcmd          jmap      keytool     rmiregistry   wsimport

[fdemeloj@fdemeloj bin]$ ./jstack --help
Usage:
    jstack [-l] <pid>
        (to connect to running process)
    jstack -F [-m] [-l] <pid>
        (to connect to a hung process)
    jstack [-m] [-l] <executable> <core>
        (to connect to a core file)
    jstack [-m] [-l] [server_id@]<remote server IP or hostname>
        (to connect to a remote debug server)

Options:
    -F  to force a thread dump. Use when jstack <pid> does not respond (process is hung)
    -m  to print both java and native frames (mixed mode)
    -l  long listing. Prints additional information about locks
    -h or -help to print this help message

Answer 4

It happpens to me yesterday, the reason appears to be the java process abnormal， ps -ef |grep java |grep xxx can not get the specific process I want. check the java process state first ,hope that helps

Answer 5

This seems to be caused by a JDK version mismatch. Make sure your running Java process was started using the same JDK as your jmap / jstack /etc. utility.

Make sure a security update or such didn't update your JDK in the meantime.

Answer 6

I think that error may be occurred due to a wrong process id that was attached to the command, You have to make sure firstly that the same version of Java is used between your compiled project and the installed JVM version itself then make sure that the correct process id is being passed to the used command whether "jstack" or "jmap", You can use the below command to find the correct process id, It's most likely the value in the second output column,

ps -fu jboss | grep java | grep wildfly