I tried to write into Json format in Java, but encountered NullPointerException when the file size is >1GB. Can anyone helps me to fix this issue?
The code keeps generating Json files, and the size of the files keep increasing. Once the file size > 1GB, the code throws exception as shown below. I used different data set for testing, so I don't think it is the data issue. My guess is that there is a size limit for Gson.toJson in Java.
My code is:
private HashMap<String,HashSet<Token>> tokenCounter = new HashMap<String,HashSet<Token>>();
....
private void writeToFile(){
try {
PrintWriter out = new PrintWriter(outputFileName);
out.println(gson.toJson(tokenCounter));
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
The exception it throws is:
java.lang.NullPointerException
at java.lang.String.<init>(String.java:301)
at java.lang.StringBuffer.toString(StringBuffer.java:790)
at java.io.StringWriter.toString(StringWriter.java:204)
at com.google.gson.Gson.toJson(Gson.java:481)
at com.google.gson.Gson.toJson(Gson.java:460)
at com.ebay.classification.discovery.DailyDiscovery.writeToFile(DailyDiscovery.java:181)
at com.ebay.classification.discovery.DailyDiscovery.run(DailyDiscovery.java:169)
at com.ebay.classification.discovery.TestDailyDiscoveryContinue.run(TestDailyDiscoveryContinue.java:142)
at com.ebay.classification.discovery.TestDailyDiscoveryContinue.main(TestDailyDiscoveryContinue.java:245)
Posted as an answer to get around formatting issues in comments.
An array of 2^30 char would be 2^31 bytes. As a single string, this is huge! The obvious question that needs to be asked is why you have the code:
PrintWriter out = new PrintWriter(outputFileName);
out.println(gson.toJson(tokenCounter));
out.close();
This can easily be written as:
FileWriter out = new FileWriter(outputFileName);
gson.toJson(tokenCounter, out);
out.flush();
out.close();
This would have no significant memory impact, and would be much faster.
This does not answer the question why you get the NPE in a large StringWriter, but, frankly, what you are doing is absurd....
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.