需要一种更有效的方法来从数据库中检索多个日期系列

Question

I have a database table that includes TRADE_DATE, CURRVAL, and ITEM fields. I first have two arrays/lists: arrVars (strings), dates (dates). Each string in arrVars represents an ITEM for which I need to retrieve the CURRVALs for each TRADE_DATE in dates. I'm new to Python, and I'm certainly no expert w/ databases, and I'm sure there are ways to speed my code up.

First part is just creating the dates list from my first db connection. I'm just iterating through each row and appending it into the dates list. Is there a better way?

i = 0
for row in cursor.fetchall():
    dates.append(row[0])
    i+=1

Second, I'm looping through each ITEM in arrVars and then looping through each TRADE_DATE in dates to create each array of CURRVALs and put the arrays into a matrix. This is pretty damn slow, so I'm hoping there is a better way as well.

M = []
dtFormat = '%Y/%m/%d'
for item in arrVars:
    tmp = []
    for dt in dates:
        strSQL = "SELECT CURRVAL FROM tblGanData WHERE ITEM = '" + item + "' AND TRADE_DATE = #" + dt.strftime(dtFormat) + "#"
        cursor.execute(strSQL)
        tmp.append(cursor.fetchone()[0])
    M.append(tmp)

Thank you!!

Answer 1

For the first bit, you might want do to something like this:

dates = [row[0] for row in cursor.fetchall()]

But I'd be very interested in seeing the SQL statement that you're using for that cursor.

select some_date from my_table

is going to be faster than

select * from my_table

(How much faster depends on how many rows you are getting back and the speed of the network connection between your client and server.)

For your second part, you're executing one query (with the full round-trip cost) for each Item/Date combination.

So maybe something like this

# build a list of all the dates
dates_str = ",".join(['#' + dt.strftime(dtFormat) + "#" 
                      for dt in dates])

# build a list of all the items
items_str = ",".join(["'" + item + "'" for item in items])

# run one SQL query that gets everything
cursor.execute("""
    select item, trade_date, currval
    from tblGanData
    where item in (%s)
    and trade_date in (%s)
    order by item, trade_date
""" % (items_str, dates_str))

You will have to do some logic when fetching the values to turn the list of currvals into a matrix, let me know if you need help with that.

PS How many items/dates are we talking about here?