Java Glassfish 4 crashes with “double free or corruption (fasttop)”

Question

I've been running Glassfish 4 on my development environment

Windows
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

Everything works perfectly.

Last week I deployed to a Debian Linux server running:

java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1~deb7u1)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

When running in the Linux environment the application intermittently crashes. It has run for several days without a crash, then crashed several times within a few hours. When it crashes there are no error messages in the glassfish or jvm log files, the process just disappears and in one case the jvm.log contained a line that was truncated. The only clue I've found so far is that the syslog and userlog contain:

grep java *

syslog:Jan 14 13:41:19 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x00007fb6ac076730 ***
syslog.1:Jan 13 19:48:04 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x00007fb037835c90 ***
user.log:Jan 13 19:48:04 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x00007fb037835c90 ***
user.log:Jan 14 13:41:19 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x00007fb6ac076730 ***
user.log.1:Jan  8 10:19:30 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x0000000007974a90 ***
user.log.1:Jan  8 14:29:11 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x00000000082431f0 ***
user.log.1:Jan  8 14:57:19 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x00007f805f87f490 ***
user.log.1:Jan  8 18:23:42 omega-rm java: *** glibc detected *** /usr/lib/jvm/java-7-openjdk-amd64/bin/java: double free or corruption (fasttop): 0x00007eff39829ca0 ***

All the errors seem identical other than the address at the end. All of the times correspond to times when the server crashed so it seems this is the cause of the disappearing server.

The application in question is a document storage system that accepts documents in multiple formats and stores them in MongoDB. Where possible it also renders the images to JPG format.

It uses Apache PDF Box, and Java Advanced Imaging to do the rendering of JPGs. It runs Spring Framework, SpringData MongoDB and Spring Security. It does occasional database access using jtds however that is infrequent and I'm fairly sure that no database activity was happening at the time of the crashes. Image reprocessing had recently happened but had completed successfully at the time of most of the crashes (not verified for all but verified in detail for the most recent crash and every image had been generated and saved for the most recently saved documents). The crash happened 50 seconds after the most recent document had been uploaded.

Virtually all the discussions I've found online happened with C or C++ programs and it would make sense there. The only way I can think of for it to happen in a Java program though is either through JNI (which I am not using, maybe some of the libraries I'm using do JNI but if so I'm not aware of it) or a JVM bug.

Does anyone have any suggestions for trying to narrow down what is causing this problem?

Is there any logging or diagnostics I can turn on to get more details?

At the moment the only thing I can think is trying to run the application for a while with certain features turned off (at the moment I'm most suspicious of the PDF rendering using PDF Box) and see which feature combination is stable and which is not. I'd rather have a more definitive approach though if possible (and one that doesn't involve waiting several days to see if a test has worked!).

Answer 1

You could try installing Oracle 'official' binaries and guidance on how to do that for Ubuntu can be found here . It uses update-alternatives which is a Debian tool and hence will be available in Ubuntu to point to the Oracle JRE installation.