在JAVA中读取超大且动态嵌套的JSON文件

Question

I have a huge json file (++500mb) consists of dynamic structure of nested json file. This json was extracted to file using 'json.dump' in python. My problem is how can i read this huge json file with buffer method?

Since if i read all the strings in the same runtime it throws java heap error. My thought is i want to read the json each record and then parse it, after that continue to next record, parse it, and so on. But how can i know which one is the end of one json record. Because i can't find the seperator between each json record.

Any suggestion? Please ask if something is not clear. Thanks

Answer 1

Assuming that you can't simply increase the heap space size with -Xmx you can switch your JSON reading logic to use a SAX JSON parsers eg RapidJSON or Jackson Streaming API . Instead of storing the entire JSON body in the memory those libraries will emit an event for each encountered JSON construct:

{
  "hello": "world",
  "t": true
  ...
}

will produce below when using RapidJSON:

StartObject()
Key("hello", 5, true)
String("world", 5, true)
Key("t", 1, true)
Bool(true)
...
EndObject()