提高sqlite查询速度

Question

I have a list of numbers (actually, percentages) to update in a database. The query is very simple, I obtain the id of the items somewhere in my code, and then, I update these items in the database, with the list of numbers. See my code:

start_time = datetime.datetime.now()

query = QtSql.QSqlQuery("files.sqlite")
for id_bdd, percentage in zip(list_id, list_percentages):
    request = "UPDATE papers SET percentage_match = ? WHERE id = ?"
    params = (percentage, id_bdd)

    query.prepare(request)

    for value in params:
        query.addBindValue(value)

    query.exec_()

    elsapsed_time = datetime.datetime.now() - start_time
    print(elsapsed_time.total_seconds())

It takes 1 second to generate list_percentages, and more than 2 minutes to write all the percentages in the database. I use sqlite for the database, and there are about 7000 items in the database. Is it normal that the query takes so much time ? If not, is there a way to optimize it ?

EDIT: Comparison with the sqlite3 module from the std library:

bdd = sqlite3.connect("test.sqlite")
bdd.row_factory = sqlite3.Row
c = bdd.cursor()

request = "UPDATE papers SET percentage_match = ? WHERE id = ?"

for id_bdd, percentage in zip(list_id, list_percentages):
    params = (percentage, id_bdd)
    c.execute(request, params)

bdd.commit()
c.close()
bdd.close()

I think the QSqlQuery commits the changes at each loop lap, while the sqlite3 module allows to commit at the ends, all the different queries at the same time.

For the same test database, the QSqlQuery takes ~22 s, while the "normal" query takes ~0.3 s. I can't believe this is just a perf issue, I must do something wrong.

Answer 1

You need to start a transaction , and commit all the updates after the loop.

Not tested but should be close to:

start_time = datetime.datetime.now()

# Start the transaction time
QtSql.QSqlDatabase.transaction()

query = QtSql.QSqlQuery("files.sqlite")
for id_bdd, percentage in zip(list_id, list_percentages):
    request = "UPDATE papers SET percentage_match = ? WHERE id = ?"
    params = (percentage, id_bdd)

    query.prepare(request)

    for value in params:
        query.addBindValue(value)

    query.exec_()

# commit changues
if QtSql.QSqlDatabase.commit():
   print "updates ok"

elsapsed_time = datetime.datetime.now() - start_time
print(elsapsed_time.total_seconds())

At the other hand, this question could to be a database performance issue, try to create an index on id field: https://www.sqlite.org/lang_createindex.html

You will need direct access to the database.

create index on papers (id);

Answer 2

Do you really need to call the prepare each time? to me, it seems the request doesn't change, so this "prepare" function could be moved out of loop?