SQLite very slow SELECT time

Question

I got some strange behavior of my program, and maybe you can bring some light into it.

Today i started testing some code, and realized, that a specific query was really slow (took about 2 minutes).

here the select:

select distinct table1.someName
from table1
INNER JOIN table2 ON table2.id = table1.t2_id
INNER JOIN table3 ON table1.id = table3.t1_id
INNER JOIN table4 ON Table3.id = table4.t3_id 
INNER JOIN table5 ON table5.id = table4.t5_id 
INNER JOIN table6 ON table4.id = table6.t4_id 
where t4_name = 'whatever'
and t2_name = 'moarWhatever'

and timestamp_till is null 

order by someName

So the thing is, the result is about 120 records. the INNER JOIN s reduce the amount of checks for timestamp_till is null to about 20 records on each record.

What bugs me most is, i've tested to insert the whole table table6 into a new created table and renamed timestamp_till to ende . On that table the select is done in about 0.1 seconds ...

Is timestamp_till some sort of reserved name of SQLite3? Could this be a bug in the SQLite engine? Is it my fault? oO

edit: add the EXPLAIN QUERY PLAN output...

When querying with the and timestamp_till is null he gives:

0|0|4|SEARCH TABLE table5 USING COVERING INDEX sqlite_autoindex_table5 (t4_name=?) (~1 rows)
0|1|3|SEARCH TABLE table4 USING INDEX table4.fk_table4_1_idx (t5_id=?) (~10 rows)
0|2|2|SEARCH TABLE table3 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0|3|0|SEARCH TABLE table1 USING INTEGER PRIMARY KEY (rowid=?) (~1rows)
0|4|1|SEARCH TABLE table2 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0|5|5|SEARCH TABLE table6 USING INDEX table6.fk_table6_ts_till (timestamp_till=?) (~2 rows)
0|0|0|USE TEMP B-TREE FOR GROUP BY
0|0|0|USE TEMP B-TREE FOR DISTINCT

and the fast one is:

   select distinct table1.someName
    from table1
    INNER JOIN table2 ON table2.id = table1.t2_id
    INNER JOIN table3 ON table1.id = table3.t1_id
    INNER JOIN table4 ON Table3.id = table4.t3_id 
    INNER JOIN table5 ON table5.id = table4.t5_id 
    INNER JOIN table6 ON table4.id = table6.t4_id 
    where t4_name = 'whatever'
    and t2_name = 'moarWhatever'    
    order by someName

and its result:

0|0|4|SEARCH TABLE table5 USING COVERING INDEX sqlite_autoindex_table5_1 (t4name=?) (~1 rows)
0|1|3|SEARCH TABLE table4 USING INDEX table4.fk_table4_1_idx (t5_id=?) (~10 rows)
0|2|2|SEARCH TABLE table3 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0|3|0|SEARCH TABLE table1 USING INTEGER PRIMARY KEY (rowid=?) (~1rows)
0|4|1|SEARCH TABLE table2 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0|5|5|SEARCH TABLE table6 USING COVERING INDEX sqlite_autoindex_table6_1 (id=?) (~10 rows)
0|0|0|USE TEMP B-TREE FOR GROUP BY
0|0|0|USE TEMP B-TREE FOR DISTINCT

with the test-table that is a copy of table6

0|0|4|SEARCH TABLE table5 USING COVERING INDEX sqlite_autoindex_table5_1 (name=?) (~1 rows)
0|1|3|SEARCH TABLE table4 USING INDEX table4.fk_t5_idx (t5_id=?) (~10 rows)
0|2|2|SEARCH TABLE table3 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0|3|0|SEARCH TABLE table1 USING INTEGER PRIMARY KEY (rowid=?) (~1rows)
0|4|1|SEARCH TABLE table2 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0|5|5|SEARCH TABLE test USING INDEX test.fk_test__idx (id=?) (~2 rows)
0|0|0|USE TEMP B-TREE FOR GROUP BY
0|0|0|USE TEMP B-TREE FOR DISTINCT

---

create script for test

CREATE TABLE "test"(
  "id" INTEGER NOT NULL,
  "t12_id" INTEGER NOT NULL,
  "value" DECIMAL NOT NULL,
  "anfang" INTEGER NOT NULL,
  "ende" INTEGER DEFAULT NULL,
  PRIMARY KEY("id","t12_id","anfang"),
  CONSTRAINT "fk_test_t12_id"
    FOREIGN KEY("t12_id")
    REFERENCES "table12"("id"),
  CONSTRAINT "fk_test_id"
    FOREIGN KEY("id")
    REFERENCES "id_col"("id"),
  CONSTRAINT "fk_test_anfang"
    FOREIGN KEY("anfang")
    REFERENCES "ts_col"("id"),
  CONSTRAINT "fk_test_ende"
    FOREIGN KEY("ende")
    REFERENCES "ts_col"("id")
);
CREATE INDEX "test.fk_test_idx_t12_id" ON "test"("t12_id");
CREATE INDEX "test.fk_test_idx_id" ON "test"("id");
CREATE INDEX "test.fk_test_anfang" ON "test"("anfang");
CREATE INDEX "test.fk_test_ende" ON "test"("ende");

soo long zai

Answer 1

A first note: SQLite will use only 1 index in its query. Never more (with the current version).

Thus, here is what SQLite is doing:

• Slow query: use the index on timestamp_till
• Fast query (no timestamp_till): use the (auto) index on table6.id .

I see two workarounds.

You could use a subquery:

select distinct SomeName FROM
(
   select table1.someName as "SomeName", timestamp_till
   from table1
   INNER JOIN table2 ON table2.id = table1.t2_id
   INNER JOIN table3 ON table1.id = table3.t1_id
   INNER JOIN table4 ON Table3.id = table4.t3_id 
   INNER JOIN table5 ON table5.id = table4.t5_id 
   INNER JOIN table6 ON table4.id = table6.t4_id 
   where t4_name = 'whatever'
   and t2_name = 'moarWhatever'
) Q
where timestamp_till is null 
order by SomeName;

Or you can drop your index on timestamp_till , if you don't need it elsewhere.

There is also perhaps some speed gains to be made by reordering your joins. Usually the smallest table first is faster, but this can vary greatly.