JVM crashes with EXCEPTION_ACCESS_VIOLATION when using Instrumentation API

Question

I am trying to use a ClassFileTransformer with the Instrumentation API . However, my JVM crashes before my premain method finishes.

I am using javassist to manipulate the byte-code of loaded classes.

My ClassFileTransformer looks like this:

import java.io.File;
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.nio.file.Files;
import java.security.ProtectionDomain;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

import javassist.ClassPool;
import javassist.CtBehavior;
import javassist.CtClass;
import javassist.Modifier;

public class JSGMain implements ClassFileTransformer {
    public static void premain(String agentArguments, Instrumentation instrumentation) {
        System.out.println("premain");
        instrumentation.addTransformer(new JSGMain());
        System.out.println("end premain");
    }

    public byte[] transform(
            ClassLoader loader, String className,
            Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
            byte[] classfileBuffer) throws IllegalClassFormatException 
    {
        System.out.println("class="+className);
        if (className == null) {
            return classfileBuffer;
        }
        className = className.replace('/', '.');
        // Check if class should be excluded / included
        if (isExcluded(className) || !isIncluded(className)) {
            return classfileBuffer;
        }
        // If the class object is loaded but an exception is thrown we must detach it in the finally block
        CtClass classObj = null;
        // If we do not make any modifications we return the original byte code
        byte[] result = classfileBuffer;

        try {
            ClassPool pool = ClassPool.getDefault();
            CtClass exceptionClass = pool.get("java.lang.Exception");
            // Load class file
            classObj = pool.makeClass(new java.io.ByteArrayInputStream(classfileBuffer));
            // Dont manipulate interfaces
            if (!classObj.isInterface()) {
                // Manipulate class methods
                CtBehavior[] methods = classObj.getDeclaredBehaviors();
                for (int i = 0; i < methods.length; i++) {
                    // Dont manipulate abstract or native methods
                    if (canTransformMethod(methods[i])) {
                        System.out.println("method="+methods[i].getName());
                        addMethodCallback(className, methods[i], exceptionClass);
                    }
                }
                // If class was modified we return the new byte code
                result = classObj.toBytecode();
            }
        } catch (Exception e) {
            System.err.println("className="+className);
            e.printStackTrace();
        } finally {
            // We might not have modified the class at all
            if (classObj != null) {
                System.out.println("beforeDetach="+classObj.getName());
                classObj.detach();
                System.out.println("afterDetach="+classObj.getName());
            }
        }
        // Return either original byte code or modified byte code
        return result;
    }

    private boolean canTransformMethod(CtBehavior method) {
        return !isAbstract(method) && !isNative(method);
    }

    private boolean isAbstract(CtBehavior method) {
        return (method.getModifiers() & Modifier.ABSTRACT) == Modifier.ABSTRACT;
    }

    private boolean isNative(CtBehavior method) {
        return (method.getModifiers() & Modifier.NATIVE) == Modifier.NATIVE;
    }

    private boolean isStatic(CtBehavior method) {
        return (method.getModifiers() & Modifier.STATIC) == Modifier.STATIC;
    }

    private void addMethodCallback(String className, CtBehavior method, CtClass exceptionClass) throws Exception {
        String methodName = method.getName();
        String beforeCallback;
        String afterCallback;
        if (method.getMethodInfo().isConstructor()) {
            beforeCallback = getClass().getName()+".beforeConstructor(\""+className+"\");";
            afterCallback = getClass().getName()+".afterConstructor(this);";
        } else if (isStatic(method)) {
            beforeCallback = getClass().getName()+".beforeStaticMethod(\""+className+"\", \""+methodName+"\");";
            afterCallback = getClass().getName()+".afterStaticMethod(\""+className+"\", \""+methodName+"\");";
        } else {
            beforeCallback = getClass().getName()+".beforeMethod(this, \""+methodName+"\");";
            afterCallback = getClass().getName()+".afterMethod(this, \""+methodName+"\");";
        }
        method.insertBefore(beforeCallback);
        method.insertAfter(afterCallback);

        String catchCallback = "{"+getClass().getName()+".exception(e); throw e;}";
        method.addCatch(catchCallback, exceptionClass, "e");
    }
}

It works for most programs, but when I try to use this class loader on an AWT / Swing application my JVM crashes with the following log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000005e592f19, pid=2456, tid=5044
#
# JRE version: Java(TM) SE Runtime Environment (8.0_05-b13) (build 1.8.0_05-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.5-b02 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V  [jvm.dll+0x132f19]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

(The log is too long for me to post) Of the log, this line grabbed my attention:

Event: 3.592 Executing VM operation: ParallelGCFailedAllocation

This might be the cause for my error, but I dont know how to interprete it or how to fix it.

This is the way I am starting my program:

java -Xbootclasspath/p:JSG.jar -javaagent:JSG.jar -jar App.jar

Thanks for your time and help.

Answer 1

I found out what the problem was: I was running out of memory.

The premain method did end, but before the text was printed the JVM crashed already.

It turns out AWT and Swing call buttloads of methods, I was doing some operations on any method being called, and those were simply too much for the system. Why it didnt fail with an OutOfMemoryException instead is not known to me though.