Reading a large dataset efficiently - Android

Question

I am developing an Android app with offline search functionality. This requires me to read in a dictionary file that has approximately 170,000 entries.

I am facing severe performance issues and initially thought it was due to my code having String.match(regex) looping through the ArrayList which I had read the data into.

However, digging deeper, I found that the main issue was actually data I/O. It took ~10,000 ms just to read the dictionary file in via BufferedReader + InputStream , without performing any searches. I've tried other means of reading data in such as using a StringBuilder but they don't seem to help much.

What are some possible solutions to this problem?

Some points to consider:

• The searches are recursive
• The app has to stay offline
• The entries are basically lines of text, albeit in an East Asian language (increasing the difficulty of the search, since entire sentences can be a single string.)

The standard code I was using for reading in the data was:

InputStream is = getResources().openRawResource(R.raw.data);     
BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
     while(br.readLine() != null){
         blahblah....
        }

Answer 1

I agree with Reuben L. I would use sqlite database instead of parsing text file.

In one of my app I put all villages and cities in Czech Republic (over 200000 records) into database for fast offline search. It could find list of cities based on name prefix in 1 sec max.

To have it even faster I introduced db indexes. I added one column and filled there the first letter of city name. Then I could search the cities by prefix like this:

select * from cities where firstLetter = ? and cityName like ?;

It improved performance more than twice.