在Sqlite中提高查询速度

Question

I am a very newbie in using python and sqlite. I am trying to create a script that reads a data from a table (rawdata) and then performs some calculations which is then stored in a new table. I am counting the number race that a player has won before that date at a particular track position and calculating the percentage. There are 15 track positions in total. Overall the script is very slow. Any suggestions to improve its speed. I have already used the PRAGMA parameters.

Below is the script.

for item in result:
        l1 = str(item[0])
        l2 = item[1]
        l3 = int(item[2])

        winpost = []
        key = l1.split("|")
        dt = l2

        ###Denominator--------------
        cursor.execute(
            "SELECT rowid FROM rawdata WHERE Track = ? AND Date< ? AND Distance = ? AND Surface =? AND OfficialFinish=1",
            (key[2], dt, str(key[4]), str(key[5]),))
        result_den1 = cursor.fetchall()
        cursor.execute(
            "SELECT rowid FROM rawdata WHERE Track = ? AND RaceSN<= ? AND Date= ? AND Distance = ? AND Surface =? AND OfficialFinish=1",
            (key[2], int(key[3]), dt, str(key[4]), str(key[5]),))
        result_den2 = cursor.fetchall()
        totalmat = len(result_den1) + len(result_den2)

        if totalmat > 0:

            for i in range(1, 16):
                cursor.execute(
                    "SELECT rowid FROM rawdata WHERE Track = ? AND Date< ? AND PolPosition = ? AND Distance = ? AND Surface =? AND OfficialFinish=1",
                    (key[2], dt, i, str(key[4]), str(key[5]),))
                result_num1 = cursor.fetchall()
                cursor.execute(
                    "SELECT rowid FROM rawdata WHERE Track = ? AND RaceSN<= ?  AND Date= ? AND PolPosition = ? AND Distance = ? AND Surface =? AND OfficialFinish=1",
                    (key[2], int(key[3]), dt, i, str(key[4]), str(key[5]),))
                result_num2 = cursor.fetchall()
                winpost.append(len(result_num1) + len(result_num2))

            winpost = [float(x) / totalmat for x in winpost]
            rank = rankmin(winpost)
            franks = list(rank)
            franks.insert(0, int(key[3]))
            franks.insert(0, dt)
            franks.insert(0, l1)
            table1.append(franks)
            franks = []

    cursor.executemany("INSERT INTO posttable VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", table1)

Answer 1

Sending and retrieving an SQL query is "expensive" in terms of time. The easiest way to speed things up would be to use SQL functions to reduce the number of queries.

For example, the first two queries could be reduced to a single call using COUNT(), UNION, and Aliases.

SELECT COUNT(*)
 FROM
 ( SELECT rowid FROM rawdata where ...
   UNION
   SELECT rowid FROM rawdata where ...
 ) totalmatch

In this case we take the two original queries (with your conditions in place of the "...") combine them with a UNION statement, give that union the alias "totalmatch", and count all the rows in it.

Same thing can be done with the second set of queries. Instead of cycling 16 times over 2 queries (resulting in 32 calls to the SQL engine) you can replace it with one query by also using GROUP BY.

SELECT PolPosition, COUNT(PolPosition)
FROM
( SELECT PolPosition FROM rawdata WHERE ...
  UNION
  SELECt PolPosition FROM rawdata WHERE ...
) totalmatch
GROUP BY PolPosition

In this case we take the exact same query as before and group it by PolPosition, using COUNT to display how many rows are in each group.

W3Schools is a great resource for how these functions work: http://www.w3schools.com/sql/default.asp