OutOfMemory错误，但应用程序在Java中继续

Question

I have faced OutOfMemory error in my application. But the thing is my application keeps on running though it has thrown OutOfMemeory. After some two minutes later, JVM has quit and pid file got genearted throwing

java.lang.OutOfMemoryError: requested 746 bytes for jbyte in 
    /BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?.

1) Is it possible for JVM to continue running even after throwing OutOfMemory error ? 2)Also can you give me some direction to investigate on this further?

I am pasting the excerpt from log to get a better understanding.

1) There are two out of memory errors. One at 2011.12.09 16.04.09:446 and another at 2011.12.09 16.04.40:818

    2011.12.09 16.04.09:446 664849490 WARNING {HTTP@9800-168}RULEZ runtime exception [transcoding]
    java.lang.OutOfMemoryError
            at java.util.zip.Inflater.inflateBytes(Native Method)
            at java.util.zip.Inflater.inflate(Inflater.java:238)
            at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:135)
            at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
            at java.util.jar.JarVerifier$VerifierStream.read(JarVerifier.java:428)
            at java.io.FilterInputStream.read(FilterInputStream.java:66)
            at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:75)
            at javax.mail.Session.loadProvidersFromStream(Session.java:932)
            at javax.mail.Session.access$000(Session.java:174)
            at javax.mail.Session$1.load(Session.java:870)
            at javax.mail.Session.loadResource(Session.java:1084)
            at javax.mail.Session.loadProviders(Session.java:889)
            at javax.mail.Session.<init>(Session.java:210)
            at javax.mail.Session.getInstance(Session.java:249)
    .
    .
    .[Application continues here]
    .
    .
    .
    2011.12.09 16.04.40:818 664882092 WARNING {HTTP@9800-168}RULEZ runtime exception [transcoding]
    java.lang.OutOfMemoryError
            at java.util.zip.Inflater.inflateBytes(Native Method)
            at java.util.zip.Inflater.inflate(Inflater.java:238)
            at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:135)
            at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
            at java.util.jar.JarVerifier$VerifierStream.read(JarVerifier.java:428)
            at java.io.FilterInputStream.read(FilterInputStream.java:66)
            at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:75)
            at javax.mail.Session.loadProvidersFromStream(Session.java:932)
            at javax.mail.Session.access$000(Session.java:174)
            at javax.mail.Session$1.load(Session.java:870)
            at javax.mail.Session.loadResource(Session.java:1084)
            at javax.mail.Session.loadProviders(Session.java:889)
            at javax.mail.Session.<init>(Session.java:210)
            at javax.mail.Session.getInstance(Session.java:249)
    .
    .
    .
    .
    .

2) At Dec 9 16:07,JVM quits and created pid file as below. So, there is ~2 minutes gap between OutOFMemory error log statement and for jvm to quit and create pid file.

    $ls

    -rw-rw-r--   1 ins ins     156076 Dec  9 16:07 hs_err_pid29157.log
    -rw-------   1 ins ins 3055525888 Dec  9 16:07 core.29157


    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    # java.lang.OutOfMemoryError: requested 746 bytes for jbyte in /BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?
    #
    #  Internal Error (allocation.inline.hpp:39), pid=21129, tid=667401120
    #  Error: jbyte in /BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jni.cpp
    #
    # JRE version: 6.0_21-b06
    # Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode linux-x86 )
    # If you would like to submit a bug report, please visit:
    #   http://java.sun.com/webapps/bugreport/crash.jsp
    #

    ---------------  T H R E A D  ---------------

    Current thread (0x2f929000):  JavaThread "DRIVER:adapter:12:7" [_thread_in_vm, id=21834, stack(0x27c5b000,0x27c7c000)]

    Stack: [0x27c5b000,0x27c7c000],  sp=0x27c7a874,  free space=7e27c7c000k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V  [libjvm.so+0x6a9262]
    V  [libjvm.so+0x2b277f]
    V  [libjvm.so+0x3c0b0f]
    C  [libocijdbc10.so+0x108ff]
    C  [libocijdbc10.so+0x11788]  Java_oracle_jdbc_driver_T2CConnection_lobGetLength+0x30
    J  oracle.jdbc.driver.T2CConnection.lobGetLength(J[BI)J

    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    J  oracle.jdbc.driver.T2CConnection.lobGetLength(J[BI)J
    J  oracle.jdbc.driver.T2CConnection.length(Loracle/sql/BLOB;)J
    J  com.fh.common.database.ReadConnectionImpl.getByteArray(Ljava/lang/String;Ljava/sql/ResultSet;I)[B
    J  com.fh.common.database.ReadConnectionWrapper.getByteArray(Ljava/lang/String;Ljava/sql/ResultSet;I)[B
    J  com.fh.mr.storage.MultiJDBCStorage.fillInitiator(Lcom/fh/common/database/ReadConnection;Ljava/lang/String;Ljava/sql/ResultSet;Lcom/fh/mr/router/TransactionResponder;)Lcom/fh/mr/router/TransactionInitiator;
    J  com.fh.mr.storage.MultiJDBCStorage.addInitiators(Lcom/fh/mr/router/TransactionResponder;Ljava/util/Collection;)V
    J  com.fh.mr.storage.CacheStorage.getResponderByTransactionID(Ljava/lang/String;)Lcom/fh/mr/router/TransactionResponder;
    j  com.fh.mr.router.RoutingEngine.getResponderByTransactionID(Ljava/lang/String;)Lcom/fh/mr/router/TransactionResponder;+5
    j  com.fh.mr.router.RoutingEngine.scheduleStatusResend(Ljava/lang/String;)Z+53
    j  com.fh.mr.router.RoutingEngine.scheduleResend(Lcom/fh/mr/message/Message;I)Z+68
    j  com.fh.mr.drivers.Driver.getMessage(Ljava/lang/Object;)Lcom/fh/mr/message/Message;+260
    J  com.fh.mr.drivers.Driver$QueueWorker.run()V
    j  com.fh.common.threadpool.ThreadControl.run()V+47
    j  com.fh.common.threadpool.ThreadPool$PoolThread.run()V+467
    v  ~StubRoutines::call_stub

Thanks,
Vijay

Answer 1

Is it possible for JVM to continue running even after throwing OutOfMemory error ?

Sure. The OutOfMemory error can be caught, and even if not, it just kills the current thread. Any other threads can continue to run.

Also can you give me some direction to investigate on this further?

According to the stacktrace, your program ran out of memory while trying to open some jar file. However, that does not necessary need to be the culprit (ie the task that ate up too much memory). You need to find out what part of your program uses too much memory, and then either reduce that somehow, or allocate more memory to the JVM.

You should also watch for memory leaks which sometimes happen for long-running processes, especially when some classes get reloaded, for example. If you find that you have a server process that slowly grows in size over time, a quick fix (easier than changing code) could be to restart it once in a while.

At Dec 9 16:07,JVM quits and created pid file as below.

The JVM crashed there with an internal error. That should not happen. However, this may very well have been caused by operating for two minutes very close to the memory limit. After an OutOfMemory error, things tend to break. Do you get this crash report more often?

Answer 2

1) Is it possible for JVM to continue running even after throwing OutOfMemory error ?

Yes, after a fashion, depending on what the application is and what it is doing.

The lifecycle of an OOME is a follows:

• The application attempts to allocate some memory (a new object or array, a new thread stack, a non-heap buffer, etcera), and finds that it can't.
• (For a normal heap allocation) the GC runs ... and fails to reclaim enough memory.
• An exception is thrown on the thread that attempted to allocate the memory.

What happens next depends on how the application deals with the exception.

• If the application catches it and bails out, that is it. The same if the OOME was thrown on the "main" thread and the application doesn't catch it.
• If the application catches it and attempts to resume then ... see below.
• If the application doesn't catch it, and the thread is NOT the main thread, then the thread dies but the application keeps going.

So lets assume that we are "keeping going". Will it work? The answer is that it depends.

• Can the application detect that the thread has died and recreate a replacement?
• What was the application in the middle of doing when the OOME was thrown?
• Was it in the middle of updating a key data structure?
• Has unwinding the stack caused enough objects to become eligible for garbage collection to allow the application to make progress?
• Will the application just repeat the computation that it did before that resulted in the OOME?

Unfortunately, the application typically can't continue due to the above issues. And even when it can continue, it often will fail again fairly shortly. The root cause of the OOME in a long running application is typically a memory leak, and the best short-term cure is to restart the JVM and the application.

2)Also can you give me some direction to investigate on this further?

If you have a memory leak, you probably can't deduce anything by looking at what was happening when the OOME was thrown. The allocation that failed is unlikely to be the real cause of the problem.

So my advice would be to start by checking the application for memory leaks.

Answer 3

As others said OutOfMemoryError is an Error that can be caught and handled.

To investigate the error further you will need to use VisualVM either to

• look at memory consumption on a live application
• browse heap dump generated by java.

To generate the heap dump start your application with -XX:+HeapDumpOnOutOfMemoryError parameter, wait for the OutOfMemoryError, grab the heap dump and feed it to VisualVM.

Googling for VisualVM and OutOfMemoryError will give you all kind of usage scenarios.