如何在python中更有效地搜索大型列表？

Question

Problem: I am working with a very large data set that I need to iterate over. Every five minutes my program adds about 1300 rows of information each with 4 columns. This means that in the course of one day it gathers about 374,400 rows of information or 1,497,600 cells per day. There are 1300 rows because there are 1300 items that the program is tracking every five minutes. For example:

Item_Name       Price      Quantity_in_Stock        Maximum_Stock_Level
----------
Soap            1.00              10                     10                    
Frogs           1.25              12                     16
Pickled Yogurt  1.35               7                      8
Malodorous Ooze 6.66               6                     66

I'm trying to tally the changes over the course of the day in the stock levels of each unique item. My current technique pulls the entire data set from a mysql server. I rely upon the item name, the stock level, the maximum stock, and the observation date:

q = """SELECT Item_Name,Item_In_Stock,Item_Max,Observation_Date
    FROM DB WHERE
    Observation_Date>DATE_ADD(curdate(),INTERVAL -1 DAY) """ 


try:
    x.execute(q)
    conn.commit()
    valueValue= x.fetchall() # The entire data set
except:
    conn.rollback()

Then I iterate through each Item_Name and for each item I find all matching values:

for item in ItemNames:
     matching = [s for s in valueValue if item[0] in s] # item[0] is an item name, i.e. Soap, Frogs, Pickled Yogurt, etc.

After that, I want to find out the number of items purchased for that day. This is tricky because items are restocked and therefore I have to compare each time interval against the last to see if there is any change in stock level (I can't just compare the beginning and end):

for item in matching:
    if not tempValue:
        tempValue = item[1] #for first row, set value equal to first row

    if tempValue > item[1]: #if last row greater than current row
        buyCount = buyCount + (item[1]-tempValue) # Add the different to the buyCount (volume sold)
    tempValue = item[1] #set tempValue for next row comparison

This method works, but it is fairly slow. I've timed it at about 2.2 seconds per unique item (out of the 1300) tallying iteration. This means that the entire day takes about 50 minutes to calculate. I'd like to cut down on this time if possible. What can I do to improve on this searching and tallying function?

EDIT: I've tried letting MySQL do the work with the following code, but it is actually slower than using python to sort through it all:

for item in getnameValues: # for each item name execute the following query
    q = """SELECT Item_Name,Item_In_Stock,Item_Max,Observation_Date
       FROM DB WHERE
       Item_Name=%s and Observation_Date>DATE_ADD(curdate(),INTERVAL -1 DAY) """
 try:
    x.execute(q,item[0]) # executes the query for the current item
    conn.commit()
    valueValue= x.fetchall() 

I'm assuming I need a way to loop through all the items within MySQL, and then have it send a list of lists back to python. Right?

Answer 1

I'm sorry, in its current form this all looks very scary.

First, the results of the computation seem to depend on the time you run them. You compute something starting from yesterday up to the moment, not just for yesterday. That is records for today (inserted before you run the script), are processed today and tomorrow.

Second, you seem to iterate over the whole dataset len(item_names) times, that is 1300 times you iterate 1.5m rows! Why not do the processing in a single iteration using defaultdict or Counter ?

Third, you should better operate with integer values instead of comparing item name strings.

Answer 2

for better performance you should use mysql programming instead python programming.

if you want to have a control over each insertion to your table, it's better use trigger in mysql. And if you want at the end of (for example) each day have search or whatever you want, you'd better use cursor .

you can find a lot of stuff around both of cursor and trigger by a simple search in internet. By the way tutsplus.com, have some neat and clean tutorial about them.