写入文件时出现Java堆空间错误

Question

I am trying to write a huge data around 64000 records at a time to a file. I am getting exceptions that I attached bellow.

the code that I used to write is

Path outputpath = Paths.get("file1.json");
try (BufferedWriter writer = Files.newBufferedWriter(outputpath, StandardCharsets.UTF_8, WRITE)) {
 writer.write(jsonObject.toString());
} catch (Exception e) {
//error msg
}

Here my "jsonObject" is nothing but a json Array which contains 65000 rows .

Can you please help me to write this to my file in an efficient way ,so that I can avoid that heap Space Error .

Answer 1

You've cut your stacktrace a little bit short, but I'll assume the exception happens in jsonObject.toString().

Basically, you have to decide between two things: either allocate more memory or break the big operation into several smaller ones. Adding memory is quick and simple, but if you expect even more data in the future, it won't solve your problem forever. As others have mentioned, use -Xmx and/or -Xms on java command line.

The next thing you could try is to use a different JSON library. Perhaps the one you are using now is not particularly suited for large JSON objects. Or there might be a newer version.

As the last resort, you can always construct the JSON yourself. It's a string after all, and you already have the data in memory. To be as efficient as possible, you don't even need to build the entire string at once, you could just go along and write bits and pieces to your BufferedWriter.

Answer 2

You can try to iterate through your json object:

Iterator<String> keys = (Iterator<String>) jsonObject.keys();
while (keys.hasNext()) {
    String key = keys.next();
    JSONObject value = jsonObject.getJSONObject(key);
    writer.write(value.toString());
}

PS. You need to check your json object's structure.