与“=”运算符相比，SQLite“LIKE”运算符非常慢

Question

When I am using the LIKE operator in SQLite, it is very slow compared to when I use the = instead. It takes about 14ms with the = operator, but when I use LIKE , it takes about 440ms. I am testing this with DB Browser for SQLite . Here is the query that works fast:

SELECT re.ENTRY_ID, 
       GROUP_CONCAT(re.READING_ELEMENT, '§') AS read_element,
       GROUP_CONCAT(re.FURIGANA_BOTTOM, '§') AS furigana_bottom,
       GROUP_CONCAT(re.FURIGANA_TOP, '§') AS furigana_top,
       GROUP_CONCAT(re.NO_KANJI, '§') AS no_kanji,
       GROUP_CONCAT(re.READING_COMMONNESS, '§') AS read_commonness, 
       GROUP_CONCAT(re.READING_RELATION, '§') AS read_rel,
       GROUP_CONCAT(se.SENSE_ID, '§') AS sense_id, 
       GROUP_CONCAT(se.GLOSS, '§') AS gloss, 
       GROUP_CONCAT(se.POS, '§') AS pos, 
       GROUP_CONCAT(se.FIELD, '§') AS field,
       GROUP_CONCAT(se.DIALECT, '§') AS dialect, 
       GROUP_CONCAT(se.INFORMATION, '§') AS info 
FROM Jmdict_Reading_Element AS re LEFT JOIN 
     Jmdict_Sense_Element AS
     se ON re.ENTRY_ID = se.ENTRY_ID
WHERE re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Reading_Element WHERE READING_ELEMENT = 'example') OR 
      re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Sense_Element WHERE GLOSS = 'example')
 GROUP BY re.ENTRY_ID

The slows down when I change

WHERE re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Reading_Element WHERE READING_ELEMENT = 'example') OR 
re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Sense_Element WHERE GLOSS = 'example')

to

WHERE re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Reading_Element WHERE READING_ELEMENT LIKE 'example') OR 
re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Sense_Element WHERE GLOSS LIKE 'example')

I need to do this so that I can use wildcards eg

WHERE re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Reading_Element WHERE READING_ELEMENT LIKE 'example%') OR 
re.ENTRY_ID IN (SELECT ENTRY_ID FROM Jmdict_Sense_Element WHERE GLOSS LIKE 'example%')

Here is a link to the database itself: https://www.mediafire.com/file/hyuymc84022gzq7/dictionary.db/file

Thanks

Answer 1

Try putting a full-text index on the columns you are using.

see Full Text Indexing

Create Catalog

USE {yourDB}  
GO  
CREATE FULLTEXT CATALOG {catalogName}
WITH ACCENT_SENSITIVITY = OFF

Create Index

USE {yourDB}  
GO  
CREATE FULLTEXT INDEX ON {someTable} ({col1}, {col2})
ON catalogName

Note This is more of a convenience, but see if your collation is case insensitive so that for example an 'a' = 'A'. Normally the collation will have a ci_utf8 for example (ci = case insensitive). I do this for the user and programmer convenience.

Answer 2

FTS?

Using DB Browser for sqlite on Win 10.

• "fast" query returns 25 rows in 84ms
• "slow" query (using LIKE "example%" ) returns 33 rows in 1025ms

Created fts4 tables like so:

create virtual table jre_fts using FTS4(entry_id,reading_element);
insert into jre_fts select entry_id, reading_element from Jmdict_Reading_Element;
create virtual table jse_fts using FTS4(entry_id,gloss);
insert into jse_fts select entry_id, gloss from Jmdict_Sense_Element;

It took 7390ms, database grew from 70,296KB to 110,708KB.

Modified the WHERE like so:

 WHERE re.ENTRY_ID IN (SELECT ENTRY_ID FROM jre_fts WHERE READING_ELEMENT MATCH '^example') OR 
re.ENTRY_ID IN (SELECT ENTRY_ID FROM jse_fts WHERE GLOSS MATCH '^example')

Query returned 33 rows in 60ms.

I cannot test or analyze how FTS will work on the reading_element column, but perhapse the approach shows promise.

Answer 3

I wonder if using HAVING would speed your query:

SELECT re.ENTRY_ID, 
       GROUP_CONCAT(re.READING_ELEMENT, '§') AS read_element,
       GROUP_CONCAT(re.FURIGANA_BOTTOM, '§') AS furigana_bottom,
       GROUP_CONCAT(re.FURIGANA_TOP, '§') AS furigana_top,
       GROUP_CONCAT(re.NO_KANJI, '§') AS no_kanji,
       GROUP_CONCAT(re.READING_COMMONNESS, '§') AS read_commonness, 
       GROUP_CONCAT(re.READING_RELATION, '§') AS read_rel,
       GROUP_CONCAT(se.SENSE_ID, '§') AS sense_id, 
       GROUP_CONCAT(se.GLOSS, '§') AS gloss, 
       GROUP_CONCAT(se.POS, '§') AS pos, 
       GROUP_CONCAT(se.FIELD, '§') AS field,
       GROUP_CONCAT(se.DIALECT, '§') AS dialect, 
       GROUP_CONCAT(se.INFORMATION, '§') AS info 
FROM Jmdict_Reading_Element re LEFT JOIN 
     Jmdict_Sense_Element se
     ON re.ENTRY_ID = se.ENTRY_ID
GROUP BY re.ENTRY_ID
HAVING SUM(CASE WHEN re.READING_ELEMENT = 'example' THEN 1 ELSE 0 END) > 0 OR
       SUM(CASE WHEN se.GLOSS = 'example' THEN 1 ELSE 0 END) > 0);