Android reading a large file

Question

I have a CSV file in the assets folder with more than 10000 lines of data. I want to insert this data into a SQLite database when the database is created. I cannot include a database because it is a very generic application and the model demands a CSV file instead. I don't want to read all 10000 lines of data and insert it from the memory in one stretch. How do I accomplish the task effectively and efficiently?

Answer 1

Just insert immediately once you've read the line. So, just don't store the lines in some arraylist in memory.

Eg

while ((line = reader.readLine()) != null) {
    insert(line);
}

You might want to do this in a single transaction so that you can rollback whenever it fails halfway. In the JDBC side, you may want to consider PreparedStatement#addBatch() / #executeBatch() to execute the inserts in batches. An example can be found in this answer .

Update : as a completely different but more efficient alternative, you can also leave Java outside the story and use CSV import facility provided by SQLite .

Answer 2

在这种情况下，最简单和最有效的是从文件中一次读取20-30-50行（或者可能是100行！）并插入到数据库中，但是您还需要运行一些测试来查看是什么您可以在1中读取的最佳行数然后执行:-)

Answer 3

IMPORT filename tablename

(from http://old.nabble.com/How-can-I-load-big-file-into-a-sqlite-database--td19202500.html - like LOAD DATA in mysql).

Other: disable auto commit, auto flush; read 20 lines, commit, flush.