OutOfMemoryError while inserting data from large JSON file into MySQL database

Question

I am reading a JSON file of roughly 6 GB using Java. The goal is to import all the JSON data into a MySQL database. The program runs fine for few initial records but then throws a java.lang.OutOfMemoryError with message Java heap space error . I am using the JsonParser class to read the file and using nextToken() to read the next JSON token. The system on which the program is running has 8 GB RAM.

How do I insert all the data into the the MySQL database from the JSON dataset?

Answer 1

The fact that the system has 8 GB of RAM doesn't necessarily mean that the JVM can allocate that amount of memory. The total space the JVM depends on several factors, including operating system, Java configuration and JVM overhead.

I'm guessing that you are reading the file at once, which requires the JVM to allocate roughly 6 GB of memory in order to store all bytes into memory. The JVM fails to allocate that amount of memory and throws an OutOfMemoryError .

If the abovementioned assumption is correct, then you might want process the input stream instantly while reading it. Jackson and Gson might do the job for you.

• This question has several answers which might be useful to read.
• Javarevisited.blogspot.nl provides a tutorial to process the JSON file while reading it, using the Jackson Streaming API.

More about the Java heap size: http://javarevisited.blogspot.nl/2011/05/java-heap-space-memory-size-jvm.html .