ObjectOutputStream - Object exceeding 1GB causes java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Question

I am trying to write file a Collection (ArrayList) of third party Externalizable class instances (Drools KnowledgePackage) using ObjectOutputStream in Java 7. If I limit the KnowledgePackage sizes so the resulting file is <= 1GB all is well. If I let the instance grow a fraction further such that (I believe) the file would be >1GB then I get the failure below.

The code looks like this:

Collection<KnowledgePackage> kpkgs = kbuilder.getKnowledgePackages();

ObjectOutputStream out = new ObjectOutputStream( new FileOutputStream(packageFileName) );
out.writeObject( kpkgs );
out.close();

And the error looks like this:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876)
        at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        at org.drools.rule.Package.writeExternal(Package.java:164)
        at org.drools.definitions.impl.KnowledgePackageImp.writeExternal(KnowledgePackageImp.java:161)
        at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        at java.util.ArrayList.writeObject(ArrayList.java:742)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        ... <my code>

Increasing the heap size does not make any difference. It seems something else if going on here.

I believe the cause is the internal byte array management of ObjectOutputStream. According to https://bugs.openjdk.java.net/browse/JDK-6991552 and https://bugs.openjdk.java.net/browse/JDK-6464834 the array size is grown by double + 1 each time existing array is exhausted. That means when the array reaches >=1GB it can't grow any further without blowing the (2^31)-1 maximum array size of Java.

Is there a workaround or alternative approach to writing these objects so I can at least output up to 2GB, and ideally an indefinite size. Perhaps an alternative method for writing and reading such large objects exists?

Have tried HotSpot 1.7.0_51 and OpenJDK 1.7.0_45 with the same results. In case its relevant, Drools version is 5.5.0Final

Many thanks

Answer 1

The issue isn't the jdk, it's Drools. If you look at the stack trace, the issue is that the object is being serialized to a ByteArrayOutputStream. the jdk isn't doing that, that is how the org.drools.rule.Package.writeExternal method is implemented: http://grepcode.com/file/repo1.maven.org/maven2/org.drools/drools-core/5.4.0.Final/org/drools/rule/Package.java#Package.writeExternal%28java.io.ObjectOutput%29 .

You should file a bug with drools (regarding serializing larger rule packages).

Alternately, it looks as if you use a DroolsObjectOutputStream, then it skips the secondary in-memory serialization and uses the given stream directly. This might solve your problem (assuming you can use the DOOS for your use case).

Answer 2

I usually use an alternative serialization mechanism for large, complex objects such as JSON or XML. However, rather than rewriting your code around to use JSON and XML you can keep it by changing the signature of the big class to java.io.Exernalizable and implement readExternal() and writeExternal() .