java.nio.DirectByteBuffer 尝试在 GC 中释放内存时 JVM 崩溃

Question

When I was running a Apache Spark job, which few lines of input data, the executor JVM crashed due to free java.nio.DirectByteBuffer in the GC:

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  sun.misc.Unsafe.freeMemory(J)V+0
j  java.nio.DirectByteBuffer$Deallocator.run()V+17
J 1537 C1 java.lang.ref.Reference.tryHandlePending(Z)Z (115 bytes) @ 0x00007f082d519f94 [0x00007f082d5199c0+0x5d4]
j  java.lang.ref.Reference$ReferenceHandler.run()V+1
v  ~StubRoutines::call_stub

And there was no memory pressure:

Heap:
par new generation   total 153344K, used 17415K [0x0000000738000000, 0x0000000742660000, 0x0000000742660000)
eden space 136320K,   1% used [0x0000000738000000, 0x00000007381955c8, 0x0000000740520000)
from space 17024K,   92% used [0x0000000740520000, 0x000000074148c778, 0x00000007415c0000)
to   space 17024K,    0% used [0x00000007415c0000, 0x00000007415c0000, 0x0000000742660000)
concurrent mark-sweep generation
total 2057856K, used 76674K [0x0000000742660000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace       used 49890K, capacity 50454K, committed 50540K, reserved 1093632K
class space     used 6821K, capacity 6995K, committed 7056K, reserved 1048576K

Full hs_err file: http://www.evernote.com/l/AAQu5abObUND5KFJbFNO9RpVfLQlBiwX6gg/

Answer 1

You tried to play around with memory using sun.misc.Unsafe api's that resulted in segmentation fault, hence jvm crashed.

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)

j sun.misc.Unsafe.freeMemory(J)V+0 j java.nio.DirectByteBuffer$Deallocator.run()V+17 J 1537 C1 java.lang.ref.Reference.tryHandlePending(Z)Z (115 bytes) @ 0x00007f082d519f94 [0x00007f082d5199c0+0x5d4] j java.lang.ref.Reference$ReferenceHandler.run()V+1 v ~StubRoutines::call_stub`

Answer 2

With @the8472 's suggestion, I tried BTrace and figured out the DirectByteBuffer are comming from default Kyro serializer. So I added my own Kyro serializer to deal with avro data and everything is fine now.