JSON parsing for huge data

Question

I have to download more than 40,000 json records from my web service, but it is taking so long time to download. The downloaded data is storing into a string, but I am not able parse that string. Please tell me a solution to solve this problem.

URL: https://buzoonga.co.uk/appapi/contacts.php?user_id=531

public JSONObject getJSONFromUrl(String url) {

    // Making HTTP request
    try {
        // defaultHttpClient
        DefaultHttpClient httpClient = new DefaultHttpClient();
        HttpPost httpPost = new HttpPost(url);

        HttpResponse httpResponse = httpClient.execute(httpPost);
        HttpEntity httpEntity = httpResponse.getEntity();
        is = httpEntity.getContent();           

    } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
    } catch (ClientProtocolException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }

    try {
        BufferedReader reader = new BufferedReader(new InputStreamReader(
                is, "iso-8859-1"), 8);
        StringBuilder sb = new StringBuilder();
        String line = null;
        while ((line = reader.readLine()) != null) {
            sb.append(line + "\n");
        }
        is.close();
        json = sb.toString();
    } catch (Exception e) {
        Log.e("Buffer Error", "Error converting result " + e.toString());
    }

    // try parse the string to a JSON object
    try {
        jObj = new JSONObject(json);
    } catch (JSONException e) {
        Log.e("JSON Parser", "Error parsing data " + e.toString());
    }

    // return JSON String
    return jObj;

}  

Answer 1

check out gzip. excellent text compressor. you are going to remove ~70% of the size :) you just have to gzip it serverside and unzip it clientside. if this overhead increases the time but the download is a third of the time, you might win in the end.

Also consider designing a better schema to reduce the size of your jsons. Like, instead of "animalBodyParts" put "abp" or something. you have not put an example of your jsons so I am just guessing.

Answer 2

You can use the JSONTokener class and pass the InputStream

jObj = new JSONObject(new JSONTokener(httpEntity.getContent()));

To reduce the time to download you need to support gzipped downloads on the server side or change your JSON structure to a more compact array based format