How do I generate a heap dump for my JBoss process without jhat or jstat?

Question

I'm using JBoss 7.1.3 with the following Java, Linux versions …

[dave@mymachine ~]$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
[dave@mymachine ~]$ uname -a
Linux mymachine.mydomain.org 4.1.99-99.88.amzn1.x86_64 #1 SMP Fri Feb 5 23:44:22 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I want to figure out how much heap is being used (not the maximum amount, but what is currently being used). Sadly, on our machine I don't have access to jmap or jstat. So I wanted to do a heap dump using “kill -3”. So I first got the pid of my boss process

[dave@mymachine ~]$ ps -elf | grep jboss
1 S root      1251     1  0  80   0 - 28870 -      07:03 ?        00:00:00 /bin/sh /etc/rc3.d/S84jboss start
4 S root      1254  1251  0  80   0 - 34424 -      07:03 ?        00:00:00 runuser -s /bin/bash jboss -c ulimit -S -c 0 >/dev/null 2>&1 ; LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=/var/run/jboss-as/jbos-as-standalone.pid /usr/java/jboss/bin/standalone.sh -c standalone.xml
4 S jboss     1255  1254  0  80   0 - 28271 -      07:03 ?        00:00:00 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=/var/run/jboss-as/jboss-as-standalone.pid /usr/java/jboss/bin/standalone.sh -c standalone.xml
0 S jboss     1256  1255  0  80   0 - 28272 -      07:03 ?        00:00:00 /bin/sh /usr/java/jboss/bin/standalone.sh -c standalone.xml
0 S jboss     1289  1256  3  80   0 - 603908 -     07:03 ?        00:18:26 /usr/java/default/bin/java -D[Standalone] -server -XX:+UseCompressedOops -Xms256m -Xmx1024m -XX:MaxPermSize=512m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -Dorg.jboss.as.logging.per-deployment=false -Dorg.jboss.boot.log.file=/usr/java/jboss/standalone/log/boot.log -Dlogging.configuration=file:/usr/java/jboss/standalone/configuration/logging.properties -jar /usr/java/jboss/jboss-modules.jar -mp /usr/java/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/java/jboss -Djbos.server.base.dir=/usr/java/jboss/standalone -c standalone.xml
0 S 602      21746 20632  0  80   0 - 27615 pipe_w 16:46 pts/1    00:00:00 grep --color=auto jboss

and then I ran “kill -3” …

[dave@mymachine ~]$ sudo kill -3 1289 1256

but nothing prints out and no file is generated in the current directory. How do I generate a heap dump with my current constraints?

Answer 1

I suspect that the main reason that you do not have access to the excellent serviceability tools like jmap , jstack etc. is that they are not part of the openjdk-6-jre , or openjdk-7-jre Linux packages (I am assuming debian distro here). I do not understand why that should be the case. In my opinion, these tools should be a part of the JRE and not the JDK . Here's the proof:

$> apt-file list openjdk-6-jdk | grep jmap
openjdk-6-jdk: /usr/lib/jvm/java-6-openjdk-amd64/bin/jmap
openjdk-6-jdk: /usr/lib/jvm/java-6-openjdk-amd64/man/ja_JP.eucJP/man1/jmap.1.gz
openjdk-6-jdk: /usr/lib/jvm/java-6-openjdk-amd64/man/man1/jmap.1.gz
$> apt-file list openjdk-6-jre | grep jmap
<nothing>

Anyway, to get these tools, you should consider using the JDK and install them using sudo apt-get install openjdk-6-jdk . This should get you not only the latest JDK 6 build (which, btw, very very old). Then, you should get the access to the tools that give us the heap dump for later analysis.

Heap dump is very different from the thread dump which shows you what the JVM threads are doing. One way to get the thread dump is by sending the QUIT signal to JVM process (using, like you said: kill -3 ). This will show the thread dump wherever the JVM's stdout is redirected. An (almost) equivalent tool is jstack , which is, again, a part of the JDK package, and not the JRE package.

But thread dump is not what you need. You need heap dump. So, you need jmap .

Short of this, you can estimate the actual memory being used by the JVM using the indispensable Linux top command. Make sure that you analyze the output correctly to get an estimate of how much heap is being used by the JVM.

If you are experiencing an OutOfMemoryError in your JVM, then you should consider starting it with -XX:+HeapDumpOnOutOfMemoryError so that when OOME is thrown, at least you can get a heap dump. But this way, you can get the heap dump only after you run out of heap space!

Answer 2

set -XX:HeapDumpPath=mpath/heapdump to a writable directory or file on java's command line for starting up JBOSS. This path/file needs to be writable for the user JBOSS is running under and then try the kill -3 option again.

http://five.agency/java-heap-dump/

You could also problematically do a heap dump using the HotSpotDiagnosticMXBean from your application also.

see class to dump heap