Is the Python sqlite3 module bugged and slow when it comes to SQL queries that start with comments?

Question

I noticed that my sqlite3 query takes 375 times longer if the SQL query starts with a comment using the --comment format. Is this normal behavior? Is it a bug in the built-in sqlite3 module?

import pathlib
import sqlite3
import timeit


def test_do(sql):
    db_path = pathlib.Path("./test.sqlite3")
    con = sqlite3.connect(db_path)
    cursor = con.cursor()

    table_sql: str = f"""
        CREATE TABLE IF NOT EXISTS test (
            id INTEGER PRIMARY KEY)"""

    cursor.execute(table_sql)

    for i in range(1, 43000):
        cursor.execute(sql, [i])

    con.commit()
    db_path.unlink()


sqlslow = f"""
    --comment
    INSERT INTO "test" ("id") VALUES (?)
    """

sqlfast = f"""
    INSERT INTO "test" ("id") VALUES (?)
    """

starttimeslow = timeit.default_timer()
test_do(sqlslow)
print(f"sqlslow: {timeit.default_timer() - starttimeslow}")

starttimefast = timeit.default_timer()
test_do(sqlfast)
print(f"sqlfast: {timeit.default_timer() - starttimefast}")

Results:

sqlslow: 21.521265994
sqlfast: 0.05736106100000171

Edit: I found the same behavior with /* */ style comments.

Answer 1

On the surface it would certainly seem that the performance is related to the comment and I see similar results with sqlite3 (20 secs and 0.05 secs). But I think it's a little more than that.

I replaced sqlite3 in your code with the apsw package, which has different transaction handling, and when using that, I saw 20 secs for both tests. Note that apsw doesn't automatically begin a transaction for you, which sqlite3 does.

Then I explicitly included a transaction begin/commit around the 43k inserts, and both times it took 0.05 secs. This makes sense because this code should run faster within a transaction.

This all suggests to me that sqlite3 is using a transaction in the one case (the single line INSERT) and not in the other case (the multi-line statement with comment and INSERT). Plus it seems that sqlite3's transaction handling, while designed to make things simpler for the user, is a little odd.

And, indeed, if I insert the following begin transaction explicitly, then the sqlite3 results are both fast (0.05 secs):

cursor.execute(table_sql)
cursor.execute("begin") # add this

for i in range(1, 43000):
    cursor.execute(sql, [i])

con.commit()

The output from the sqlite3 test is now:

sqlslow: 0.051317919000000004
sqlfast: 0.05007833699999999