How to restart JVM when it runs out of memory by using JMX monitoring?

Question

I want to create a nagios watchdog for JVM that looks when the JVM runs out of memory and restarts it.

Currently I was able to setup the JVM be allow JMX but I don't know how to detect OutOfMemory condition and restart it.

 /check_jmx -U service:jmx:rmi:///jndi/rmi://127.0.0.1:1100/jmxrmi -O "java.lang:type=Memory" -A "HeapMemoryUsage" -K used -I HeapMemoryUsage -J used -vvvv
 JMX OK HeapMemoryUsage.used=957414288{committed=2415984640;init=2147483648;max=2863333376;used=957414288}

https://github.com/tcurdt/nagios-check-jmx

Answer 1

If you're using Java 1.4.2 or later, this option allows you to execute a user defined command when the first OutOfMemeory exception occurs: -XX:OnOutOfMemoryError="<cmd args>;<cmd args>"

That should give you some decent options. eg you could kick off a passive check to nagios to tell it that the server is being restarted, and then kick off a shell script to stop/start your errant JVM.

Answer 2

I don't think you are going to be able to detect an out-of-memory condition using JMX. If the JVM is really at the end of it's life, JMX connections will most likely throw OOM exceptions themselves when you try to connect.

We detect high memory conditions instead of OOM. We alarm when our systems' free-memory goes below some water mark for a certain amount of time. We also have threads that run to dump per-serve metric files. Since the thread is already allocated, it can reliably dump system memory information after the JVM runs out.

We log:

// free memory
Runtime.getRuntime().freeMemory()
// current heap usage
Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()