Interpreting core-dump file stack

Question

My application causes JVM to crash randomly.

I have collected the core dumps and looked over but I need a bit help because I have a very little experience with core dumps.

Among 15 or more threads these 2 caught my attention:

Thread 1 (Thread 0xbb8f5b70 (LWP 27584)):

Thread 1 (Thread 0xbb8f5b70 (LWP 27584)):
#0 0x00a6b430 in __kernel_vsyscall ()
#1 0x0020bb11 in raise () from /lib/libc.so.6
#2 0x0020d3ea in abort () from /lib/libc.so.6
#3 0x0024b9d5 in __libc_message () from /lib/libc.so.6
#4 0x00251e31 in malloc_printerr () from /lib/libc.so.6  <---- THIS LINE
#5 0x00254823 in _int_free () from /lib/libc.so.6 
#6 0x0078a068 in _dlerror_run () from /lib/libdl.so.2
#7 0x00789d7c in dlsym () from /lib/libdl.so.2
#8 0x01397fa0 in os::dll_lookup(void*, char const*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#9 0x0136aa07 in NativeLookup::lookup_style(methodHandle, char*, char const*, int, bool, bool&, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#10 0x0136abf8 in NativeLookup::lookup_entry(methodHandle, bool&, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#11 0x0136b107 in NativeLookup::lookup_base(methodHandle, bool&, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#12 0x0136b1f7 in NativeLookup::lookup(methodHandle, bool&, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#13 0x0119a32b in InterpreterRuntime::prepare_native_call(JavaThread*, methodOopDesc*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#14 0xf477e12b in ?? ()
#15 0xf4776387 in ?? ()
#16 0xf4776387 in ?? ()
#17 0xf4776387 in ?? ()
#18 0xf4776387 in ?? ()
#19 0xf4776387 in ?? ()
#20 0xf4776387 in ?? ()
#21 0xf4776387 in ?? ()
#22 0xf4776387 in ?? ()
#23 0xf4776387 in ?? ()
#24 0xf47765fb in ?? ()
#25 0xf4773459 in ?? ()
#26 0x011a0915 in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#27 0x01396769 in os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#28 0x0119f58f in JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#29 0x0119f7ca in JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#30 0x0119f8eb in JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#31 0x0121b4c9 in thread_entry(JavaThread*, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#32 0x014b8d49 in JavaThread::thread_main_inner() () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#33 0x014b8ea3 in JavaThread::run() () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#34 0x0139d999 in java_start(Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#35 0x0037ea49 in start_thread () from /lib/libpthread.so.0
#36 0x002c3aee in clone () from /lib/libc.so.6

Thread 11 (Thread 0xf77736c0 (LWP 27570)):

Thread 11 (Thread 0xf77736c0 (LWP 27570)):
#0 0x00a6b430 in __kernel_vsyscall ()
#1 0x00385019 in __lll_lock_wait () from /lib/libpthread.so.0
#2 0x0038043e in _L_lock_731 () from /lib/libpthread.so.0
#3 0x0038034a in pthread_mutex_lock () from /lib/libpthread.so.0
#4 0x007b9b2d in _dl_open () from /lib/ld-linux.so.2
#5 0x00789c3b in dlopen_doit () from /lib/libdl.so.2
#6 0x007b5ba6 in _dl_catch_error () from /lib/ld-linux.so.2
#7 0x0078a03c in _dlerror_run () from /lib/libdl.so.2
#8 0x00789b71 in dlopen@@GLIBC_2.1 () from /lib/libdl.so.2
#9 0x0139e13e in os::dll_load(char const*, char*, int) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#10 0x0136a17f in NativeLookup::lookup_critical_style(methodHandle, char*, char const*, int, bool) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#11 0x0136a72d in NativeLookup::lookup_critical_entry(methodHandle) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#12 0x0143719c in SharedRuntime::generate_native_wrapper(MacroAssembler*, methodHandle, int, BasicType*, VMRegPair*, BasicType) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#13 0x0142476f in AdapterHandlerLibrary::create_native_wrapper(methodHandle, int) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#14 0x0101c5a9 in CompileBroker::compile_method(methodHandle, int, int, methodHandle, int, char const*, Thread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#15 0x0100aa18 in SimpleCompPolicy::method_invocation_event(methodHandle, JavaThread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#16 0x01009d5b in NonTieredCompPolicy::event(methodHandle, methodHandle, int, int, CompLevel, nmethod*, JavaThread*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#17 0x011983dd in InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#18 0x0119a982 in InterpreterRuntime::frequency_counter_overflow(JavaThread*, unsigned char*) () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#19 0xf477e525 in ?? ()
#20 0xf47ccbec in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?) <---- AND THIS LINE

Can anyone lend me a hand with understanding the key parts here?

This core dump was generated nearly 3 months ago and steps-to-reproduce are a bit inconsistent.

Extra question:

When stack entry says 0x00789b71 in dlopen@@GLIBC_2.1 () from /lib/libdl.so.2

What does the @@ mean? Can someone please explain? Google doesn't give me anything - I think it assumes @ as special char and ignores them altogether.

Answer 1

Among 15 or more threads these 2 caught my attention:

Thread 1 is the root cause of the crash; there is nothing wrong that I can see about thread 11.

Thread 1 is crashing because glibc detected heap corruption, and called abort() after reporting it to /dev/tty ...

Unfortunately, the nature of heap corruption is such that it usually causes a crash much later, in completely unrelated code. The standard advice is to run the program under Valgrind, but that advice doesn't usually work well for Java programs.

What does the @@ in dlopen@@GLIBC_2.1 mean?

It means that a function dlopen is being called, that there are several versions of this function in libdl , and the default one, maked with version GLIBC_2.1 is being called.

You can read more about GNU symbol versioning here .