Android Http Post streaming json

Question

I ran into trouble trying to send SQLite data to a web server using json (gson). Everything was fine until the table came round 6000 rows. I ran into Out of Memory errors.

In my datahelper I have:

public String DonneesToJson (SQLiteDatabase db, int chantier, int chantier_serveur )
{
    Cursor c = db.rawQuery("select * from "+P_TABLE+" where "+P_CHANTIER+"="+chantier+ " order by "+P_TIME+" desc ", null);

    List<Donnees> donnees = new ArrayList<Donnees>();
    c.moveToFirst();

    while (c.moveToNext())
    {
        Donnees d = new Donnees (
                c.getInt(c.getColumnIndex(Datahelper.P_ID)),
                chantier_serveur,
                offset,
                c.getInt(c.getColumnIndex(Datahelper.P_PLAN)),
                c.getString(c.getColumnIndex(Datahelper.P_TIME)),
                c.getInt(c.getColumnIndex(Datahelper.P_PRESSION)),
                c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR)),
                c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR_TOTALE)),
                c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_X)),
                c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_Y)),
                c.getString(c.getColumnIndex(Datahelper.P_PIEU)),
                c.getInt(c.getColumnIndex(Datahelper.P_NO_RALLONGE)),
                c.getString(c.getColumnIndex(Datahelper.P_RALLONGE)),
                c.getInt(c.getColumnIndex(Datahelper.P_MOTEUR)),
                c.getString(c.getColumnIndex(Datahelper.P_SERIE)),
                c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_A)),
                c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_B))
        );
        donnees.add(d);
    }

    c.close();

    Gson gson = new Gson();

    return gson.toJson(donnees);
}

Basically I call this like that:

String resultat = dbHelper.DonneesToJson(db,i, chantier_serveur);

HttpPost post2 = new HttpPost("http://www.zzzzzzz.com/test.php");
StringEntity se = new StringEntity(resultat);  
se.setContentEncoding( new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
post2.setEntity(se);
response = client.execute(post2);

On the server side, it's quite basic php to store data in a big sql DB and then do analysis.

ie: $decoded = json_decode($HTTP_RAW_POST_DATA,true); foreach ( $decoded as $key => $value) { $query = ... $decoded = json_decode($HTTP_RAW_POST_DATA,true); foreach ( $decoded as $key => $value) { $query = ...

While doing OOM erros, it's slow. I mean getting sql data to json is slow.

I tried to go the jackson route, faster, no out of memory error, but... it can only write to a file or a stream. I'd try to avoid writing to a file then send the file trough http post. So I decided to open an http stream to send json data and I'm stuck. I did not find any example on how to open an output stream to the web server using apache.

Any help appreciated.

Answer 1

Despite the answer from dmon to split the data into batches, you should work on other glitches. Try this code:

public String DonneesToJson (SQLiteDatabase db, int chantier, int chantier_serveur ) {
    Cursor c = db.rawQuery("select * from "+P_TABLE+" where "+P_CHANTIER+"="+chantier+ " order by "+P_TIME+" desc ", null);

    List<Donnees> donnees = new ArrayList<Donnees>();
    if (c != null) { // nullcheck as rawQuery can return null!
        Donnees d; // reuse variables for loops
        while (c.moveToNext()) { // was buggy before, read comment below code
            d = new Donnees (
                c.getInt(c.getColumnIndex(Datahelper.P_ID)),
                chantier_serveur,
                offset,
                c.getInt(c.getColumnIndex(Datahelper.P_PLAN)),
                c.getString(c.getColumnIndex(Datahelper.P_TIME)),
                c.getInt(c.getColumnIndex(Datahelper.P_PRESSION)),
                c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR)),
                c.getInt(c.getColumnIndex(Datahelper.P_PROFONDEUR_TOTALE)),
                c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_X)),
                c.getInt(c.getColumnIndex(Datahelper.P_ANGLE_Y)),
                c.getString(c.getColumnIndex(Datahelper.P_PIEU)),
                c.getInt(c.getColumnIndex(Datahelper.P_NO_RALLONGE)),
                c.getString(c.getColumnIndex(Datahelper.P_RALLONGE)),
                c.getInt(c.getColumnIndex(Datahelper.P_MOTEUR)),
                c.getString(c.getColumnIndex(Datahelper.P_SERIE)),
                c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_A)),
                c.getDouble(c.getColumnIndex(Datahelper.P_COEFF_B)));
            donnees.add(d);
        }
        c.close();
    }
    return new Gson().toJson(donnees);
}

Your current implementation is buggy and you will never get the first entry. The reason for that is your call to moveToFirst() which moves you to the first. The while loop will move the internal pointer to the second entry with moveToNext() which will result in starting with the second element and completely ignoring the first one. Always and forever...