SQLIte memory usage in C#

Question

I have a simple sqlite database with only one table and 7-8 string fields with ~2.000.000 records. Database file on disk is about 450-500mb...

When i open db file in my c# app and execute simple "select * from tbl" query, computer memory usage goes to 3gb.

Here is code i'm using to select data from database:

        DataSet ds = new DataSet();
        dataGridView1.DataSource = null;
        string s = Cstr();
        try
        {

            SQLiteConnection conn = new SQLiteConnection(s);
            SQLiteCommand command = new SQLiteCommand(conn);
            command.CommandText = "select * from log;";
            conn.Open();
            command.ExecuteReader();
            var da = new SQLiteDataAdapter(command.CommandText, conn);
            da.Fill(ds);

            dataGridView1.DataSource = ds.Tables[0];
            conn.Close();
            GC.Collect();

        }
        catch (Exception)
        {
        }

I tried same thing with mysql5 db and memory usage is normal for database of that size. edit: I tried to load same sqlite database with Delphi XE5 and it uses only ~600mb of memory.

Is there any way to avoid this?

Answer 1

This is not an authoritative answer on this but I suspect the following is the cause of what you see.

SQLite is an in-process database management system. As such, when you perform that query which selects all records in your 2,000,000 rows table you are essentially loading all that data in memory complete with the overhead that comes from whatever index data structures SQLite uses to reference the data.

Contrast that with MySQL which runs as a separate process. Most likely when you send a SELECT * FROM .. query to MySQL via the MySQL data access driver it doesn't really send back all the data at once but instead you're only getting results back gradually, as you read the next record via a DataReader.

Answer 2

I encountered this same problem before with C#.net framework 4.0 and here was my observations.

if you're doing a lot of process on the SQLite at the same time(using threads) you will create a big overhead because SQLite locks the whole database because it can only handle one process at a time.

the workaround I came to solve this problem was to dedicated myself on using transaction during CRUD operations and using prepared statements.