Serializing gigabytes of python object in Java using gson

Question

I have many hashmaps which have been serialized and dumped to a text file using python. I am trying to use gson to deserialize this data into objects.

The serialization works fine for small-sized files(~MBs) but gives error for files of giga-bytes size.

The error is:

Exception in thread "main" com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string at line 1 column 1230 path $.
at com.google.gson.Gson.fromJson(Gson.java:825)
at ReadFromJson.main(ReadFromJson.java:47)
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated string at line 1 column 1230 path $.
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1573)
at com.google.gson.stream.JsonReader.nextQuotedValue(JsonReader.java:1031)
at com.google.gson.stream.JsonReader.nextString(JsonReader.java:827)
at com.google.gson.internal.bind.TypeAdapters$13.read(TypeAdapters.java:358)
at com.google.gson.internal.bind.TypeAdapters$13.read(TypeAdapters.java:346)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.gson.Gson.fromJson(Gson.java:810)

Answer 1

You are probably encountering some limitation that is a result of the fact that Gson tries to load whole JSON structure into memory. You should consider using a JSON parser that supports streaming the data without trying to parse everything at once.

One of those is Jackson and it's Streaming API .