Simultaneous abort() in two threads

Question

I have a backtrace with something I haven't seen before. See frame 2 in these threads:

Thread 31 (process 8752):
#0  0x00faa410 in __kernel_vsyscall ()
#1  0x00b0b139 in sigprocmask () from /lib/libc.so.6
#2  0x00b0c7a2 in abort () from /lib/libc.so.6
#3  0x00752aa0 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x00750505 in ?? () from /usr/lib/libstdc++.so.6
#5  0x00750542 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x00750c65 in __cxa_pure_virtual () from /usr/lib/libstdc++.so.6
#7  0x00299c63 in ApplicationFunction()

Thread 1 (process 8749):
#0  0x00faa410 in __kernel_vsyscall ()
#1  0x00b0ad80 in raise () from /lib/libc.so.6
#2  0x00b0c691 in abort () from /lib/libc.so.6
#3  0x00b4324b in __libc_message () from /lib/libc.so.6
#4  0x00b495b6 in malloc_consolidate () from /lib/libc.so.6
#5  0x00b4b3bd in _int_malloc () from /lib/libc.so.6
#6  0x00b4d3ab in malloc () from /lib/libc.so.6
#7  0x08147f03 in AnotherApplicationFunction ()

When opening it with gdb and getting backtrace it gives me thread 1. Later I saw the weird state that thread 31 is in. This thread is from the library that we had problems with so I'd believe the crash is caused by it.

So what does it mean? Two threads simultaneously doing something illegal? Or it's one of them, causing somehow abort() in the other one?

The OS is Linux Red Hat Enterprise 5.3, it's a multiprocessor server.

Answer 1

It is hard to be sure, but my first suspicion upon seeing these stack traces would be a memory corruption (possibly a buffer overrun on the heap). If that's the case, then the corruption is probably the root cause of both threads ending up in abort .

Can you valgrind your app?

Answer 2

Looks like it could be heap corruption, detected by malloc in thread 1, causing or caused by the error in thread 31.

Some broken piece of code overwriting ao the vtable in thread 31 could easily cause this.

Answer 3

It's possible that the reason thread 31 aborted is because it trashed the application heap in some way. Then when the main thread tried to allocate memory the heap data structure was in a bad state, causing the allocation to fail and abort the application again.