Android - General advice needed on parsing a file on first run

Question

I have an app that, when you load it, it displays 6 buttons. The 6th button requires visualising of some data. This data is stored in a txt file in JSON format in my /raw folder. The data is 600kb in size, but at the moment, it takes around 3 minutes to parse and store in the database. The idea is that once its in the database, its a lot more efficient, and faster, to read and use the data, as opposed to reading it directly from the file each time.

I had originally made the app so that it downloads the data from my REST server, and then stored it in the database, but then figured it would be a lot quicker to just ship the file along with the app. However, its still pretty slow.

I am looking for advice on how to speed up this file parsing and make the user experience better. At the moment, the first time they go into this section, a ProgressDialog appears, and they have to wait a few minutes (3 minutes on my HTC Desire) for all this file parsing to be done before they can use the feature. The progress dialog box just sits there, working away until its done, and the user is unable to do anything else. Its not ideal! I had thought about just running the file processing at the start of the app in a separate thread in the background, without a progress bar, so it would appear silent to the user, however, if they select the button to go into the section that relies on this data as soon as the app loads up, then it will cause further problems, since the file processing may not have completed.

Perhaps I could add in some "instructional" popup boxes with lots of text, so that by the time they read through them, the file processing is done. However, I'm not sure what value that adds to the app. Its pretty straight-forward to use!

Can anyone give me advice on a better strategy here? Also, am I right in thinking that 3 minutes to parse 600kb of JSON is a bit too long? Perhaps my logic isn't as efficient as it should be.

Here is the start of the thread:

private boolean parseFile() {
    dialog = ProgressDialog.show(activity,
             "Progress Title",
             "Progress body",
             true);
    ReadDataThread unt = new ReadDataThread();
    unt.start();
    return true;
}

private class ReadDataThread extends Thread {
    @Override
    public void run() {
        InputStream is = activity.getResources().openRawResource(R.raw.dbdata);
        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        try {
            StringBuilder sb = new StringBuilder();
            sb.append(reader.readLine());
            sortAndStore(sb.toString());
        } catch (IOException e) {
            Log.e(Constants.TAG, "Error reading data");
            e.printStackTrace();
        }
        handler.sendEmptyMessage(0);
    }

    private final Handler handler = new Handler() {
        @Override
        public void handleMessage(Message msg) {
            if (dialog.isShowing())
                dialog.dismiss();
            Toast.makeText(activity, "Finished reading data", Toast.LENGTH_SHORT).show();
        }
    };
}

You'll notice that it calls a method called sortAndStore , which is here:

@Override
public boolean sortAndStore(String raw) {
    boolean returnCode=false;
    ContentValues initialValues = new ContentValues();

    try {
        JSONObject f = new JSONObject(raw);
        JSONArray jsonArr = f.getJSONArray("stops");
        int arrSize=jsonArr.length();
        for(int i=0; i<arrSize; i++) {              
            initialValues.put("name", jsonArr.getJSONObject(i).getString("name"));
            initialValues.put("lat", jsonArr.getJSONObject(i).getString("lat"));
            initialValues.put("lng", jsonArr.getJSONObject(i).getString("lng"));
            initialValues.put("StopNumber", jsonArr.getJSONObject(i).getString("StopNumber"));
            initialValues.put("Route", jsonArr.getJSONObject(i).getString("Route"));
            initialValues.put("Type", jsonArr.getJSONObject(i).getString("Type"));

            //Add data to DB
            DBHelper db = new DBHelper(activity);
            returnCode = db.addData(initialValues, DBHelper.DBTableName);
        }
    } catch (JSONException e) {
        returnCode=false;
        e.printStackTrace();
    }
    return returnCode;
}

I also use a DatabaseHelper class called DBHelper to handle database stuff. The DBHelper.addData() method is here:

public boolean addData(ContentValues initialValues, String tableName) {
    openDatabase();
    long returnCode = db.insert(tableName, "ID", initialValues);
    closeDatabase();
    return (returnCode == -1 ? false:true);
}

So, can anyone suggest a better strategy for this, based on the above data? Either more efficient file I/O code, or maybe some UI trick to give the user the impression that the processing is taking less time than it actually is?

Thank you.

Answer 1

I think you show show the progress with fixed area somewhere in the corner like action bar or something etc. The progress may show the progress globally in your app. Then user will know something work is going on and you can let the user some action can achieved without your data.

Answer 2

As suggested before, you can ship the app with pre-populated database. check this article, it is pretty simple and straightforward Using your own SQLite database in Android applications

Answer 3

I fixed this by simply using statements instead of separate insert statements.

For example, before the for() loop, I got the database and started the transaction. Then I went through the loop as normal. After loop, i simply closed the transaction, which enables the bulk transaction.

The inserts now take a matter of seconds!!