Handling large sqlite database on Android

Question

I am working on dedicated Android device that is used for finger print matching. The can be database on this device that has 500K records.

I have modified the ROM on these devices to increase the heap size using directive -

dalvik.vm.heapgrowthlimit=742m dalvik.vm.heapsize=812m

I am trying to pre-load about 250MB database from SQlite into memory and it takes incredible amount of time. I have implemented this code in java using the Android SQLite interfaces. It takes about 10 min for 32,000 records to be loaded, is using Android/Java interface a lost cause and I should be doing all of this in native C code?

Are there are any improvements one can do to speed cursor loading using Android Java interface?

I keep seeing these messages as Android is loading the records:

-03 09:18:53.802 D/dalvikvm( 5234): GC_FOR_ALLOC freed 18K, 3% free 305194K/313816K, paused 187ms, total 187ms
10-03 09:18:53.982 D/dalvikvm( 5234): GC_FOR_ALLOC freed 0K, 3% free 305213K/313816K, paused 178ms, total 178ms
10-03 09:18:53.982 I/dalvikvm-heap( 5234): Grow heap (frag case) to 300.238MB for 4963-byte allocation
10-03 09:18:54.163 D/dalvikvm( 5234): GC_FOR_ALLOC freed 19K, 3% free 305204K/313824K, paused 176ms, total 176ms
10-03 09:18:54.333 D/dalvikvm( 5234): GC_FOR_ALLOC freed <1K, 3% free 305222K/313824K, paused 178ms, total 178ms
10-03 09:18:54.333 I/dalvikvm-heap( 5234): Grow heap (frag case) to 300.246MB for 4389-byte allocation

Answer 1

You won't need to change most of this data, therefore you should use cdb or one of its alternatives instead. This will speed up access to the db, and you can still use SQLite to store new or changed records. Actually you might be able to do even better on load time if you use a raw memory dump format for your data but that may make search harder. If search time is an issue, consider using Lucene for storing the data in RAM.

SQL is just the wrong tool for loading this kind of data, even if you were running it on a 16 core server with 32G of RAM, I would still recommend cdb or Lucene.