在读取 Java 中的巨大 Json 响应时，超出 Memory

Question

My requirement is to read Huge Json from REST API call and process, i googled a lot to find out a feasible solution, everyone saying use JsonReader bypassing InputStream.I am doing the same but still encountered out of memory.

I am using gson library to get the data, I am not sure how to achieve this functionality, could you please guide me on this. small json response, this is working fine and fast whereas getting huge response stream I am getting out of Memory.

public void processHugeInputStream(InputStream is){
 BufferedReader br = new BufferedReader(new InputStreamReader(is));
 String line;
 List<Sample> list = new ArrayList<Sample>();
 while(line = br.readLine()!=null){
 Reader stringReader = new StringReader(line);
 JSONReader reader = new JSONReader(stringReader);
 reader.setLinent(true);
 Sample object = gson.fromJSON(reader,Sample.class);
 list.add(object);
}
}

Answer 1

It looks like you are reading the stream line by line, which is ok.

However you are allocating a List and then adding each parsed result into the list, so if you have a huge Json input stream you will end up with a huge list of parsed objects. It's probably this part that is giving you the memory error.

If you have a way of refactoring the code so that you can process each Json Object as it comes in, rather than adding them all to an array and processing later, then you can run without needing to increase the memory size of the JVM.

One way you can make this generic is to have a Consumer<Sample> function that you pass into this method, and then you can accept each Sample as it comes in.

Answer 2

Please increase your heap size in Java. Use this option with your Java command: java -Xmx6g... to increase to 6 GB

There are also more option to config your memory usage:

• Thread stack size -Xss128m – set the thread stack size to 128 megabytes.
• Young generation size -Xmn256m – set the young generation size to 256 megabytes.

Here you can get a full documentation to this topic