Java/Android: java.lang.OutOfMemoryError while building a JSON object

Question

I am importing JSON data from a public database URI http://data.seattle.gov/api/views/3k2p-39jp/rows.json and the rows go as far as 445454. Using the following code I am constructing the JSON object of the entire data.

   HttpGet get = new HttpGet(uri);
   HttpClient client = new DefaultHttpClient();
   HttpResponse response = client.execute(get);
   BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
    StringBuilder builder=new StringBuilder();
for(String line=null;(line = reader.readLine()) != null;){
      builder.append(line).append("\n");
    }
  JSONTokener jsonTokener=new JSONTokener(builder.toString());
  JSONObject finalJson=new JSONObject(jsonTokener);
  JSONArray data=finalJson.getJSONArray("data");

Because the data is too large, i am getting 03-21 03:41:49.714: E/AndroidRuntime(666): Caused by: java.lang.OutOfMemoryError pointing the source of error at buildr.append(line).append("\\n") . Is there anyway I can handle large datasets without getting memory allocation issues?

Answer 1

That JSON is huge!

You definitely need to use a streaming JSON parser. There are two out there for Android: GSON and Jackson.

GSON Streaming is explained at: https://sites.google.com/site/gson/streaming

I like how GSON explains the problem you're having:

Most applications should use only the object model API. JSON streaming is useful in just a few situations:

When it is impossible or undesirable to load the entire object model into memory. This is most relevant on mobile platforms where memory is limited.

Jackson Streaming is documented at: http://wiki.fasterxml.com/JacksonInFiveMinutes#Streaming_API_Example

Answer 2

If possible only request parts of the data - this also reduces time for network io and thus saves battery.

Otherwise you could try to not keep the incoming data in memory, but to 'stream' it onto the sd-card. When it is stored there you can then iterate over it. Most likely this will mean to use your own JSON tokenizer that does not build a full tree, but which is able to (like a SAX parser) only look at a part of the object tree at a time.

You may have a look at Jackson , which has a streaming mode, which may be applicable.

Answer 3

Streaming pull parser is the way. I recommend GSON, as this has small memory footpring (just pull parsing is about 16K , jackson is way bigger)

Your code is problematic because you allocate:

• buffer to hold all the string data coming from service
• all the JSON DOM objects

and this is slow, and gives you memory meltdown.

In case you need java objects out of your JSON data , you may try my small databinding library building on GSON (shameles self advertising off):

https://github.com/ko5tik/jsonserializer

Answer 4

I did it a bit differently, My JSON code was waiting for status, which comes towards the end. So I modified the code to return earlier.

// try to get formattedAddress without reading the entire JSON
        String formattedAddress;
        while ((read = in.read(buff)) != -1) {
            jsonResults.append(buff, 0, read);
            formattedAddress = ((String) ((JSONObject) new JSONObject(
                    jsonResults.toString()).getJSONArray("results").get(0))
                    .get("formatted_address"));
            if (formattedAddress != null) {
                Log.i("Taxeeta", "Saved memory, returned early from json") ;
                return formattedAddress;
            }               
        }
JSONObject statusObj = new JSONObject(jsonResults.toString());
        String status = (String) (statusObj.optString("status"));
        if (status.toLowerCase().equals("ok")) {
            formattedAddress = ((String) ((JSONObject) new JSONObject(
                    jsonResults.toString()).getJSONArray("results").get(0))
                    .get("formatted_address"));
            if (formattedAddress != null) {
                Log.w("Taxeeta", "Did not saved memory, returned late from json") ;
                return formattedAddress;
            }           
        }