在 pandas dataframe 上迭代 function 的最快方法

Question

I have a function which operates over lines of a csv file, adding values of different cells to dictionaries depending on whether conditions are met:

df = pd.concat([pd.read_csv(filename) for filename in args.csv], ignore_index = True)

ID_Use_Totals = {}
ID_Order_Dates = {}
ID_Received_Dates = {}
ID_Refs = {}
IDs = args.ID

def TSQs(row):

    global ID_Use_Totals, ID_Order_Dates, ID_Received_Dates

    if row['Stock Item'] not in IDs:
        pass
    else:
        if row['Action'] in ['Order/Resupply', 'Cons. Purchase']:
            if row['Stock Item'] not in ID_Order_Dates:
                ID_Order_Dates[row['Stock Item']] = [{row['Ref']: pd.to_datetime(row['TransDate'])}]
            else:
                ID_Order_Dates[row['Stock Item']].append({row['Ref']: pd.to_datetime(row['TransDate'])})
        
        elif row['Action'] == 'Received':
                
             if row['Stock Item'] not in ID_Received_Dates:
                ID_Received_Dates[row['Stock Item']] = [{row['Ref']: pd.to_datetime(row['TransDate'])}]
            else:
                ID_Received_Dates[row['Stock Item']].append({row['Ref']: pd.to_datetime(row['TransDate'])})
                                    
        elif row['Action'] == 'Use':
            if row['Stock Item'] in ID_Use_Totals: 
                ID_Use_Totals[row['Stock Item']].append(row['Qty'])
            else:
                ID_Use_Totals[row['Stock Item']] = [row['Qty']]
                                       
        else:
            pass

Currently, I am doing:

for index, row in df.iterrows():
    TSQs(row)

But timer() returns between 70 and 90 seconds for a 40,000 line csv file.

I want to know what the fastest way of implementing this is over the entire dataframe (which could potentially be hundreds of thousands of rows).

Answer 1

I'd wager not using Pandas for this could be faster.

Additionally, you can use defaultdict s to avoid having to check whether you've seen a given product yet:

import csv
import collections
import datetime

ID_Use_Totals = collections.defaultdict(list)
ID_Order_Dates = collections.defaultdict(list)
ID_Received_Dates = collections.defaultdict(list)
ID_Refs = {}
IDs = set(args.ID)
order_actions = {"Order/Resupply", "Cons. Purchase"}

for filename in args.csv:
    with open(filename) as f:
        for row in csv.DictReader(f):
            item = row["Stock Item"]
            if item not in IDs:
                continue
            ref = row["Ref"]
            action = row["Action"]
            if action in order_actions:
                date = datetime.datetime.fromisoformat(row["TransDate"])
                ID_Order_Dates[item].append({ref: date})
            elif action == "Received":
                date = datetime.datetime.fromisoformat(row["TransDate"])
                ID_Received_Dates[item].append({ref: date})
            elif action == "Use":
                ID_Use_Totals[item].append(row["Qty"])

EDIT: If the CSV is really of the form

"Employee", "Stock Location", "Stock Item"
"Ordered", "16", "32142"

the stock CSV module can't quite parse it.

You could use Pandas to parse the file, then iterate over rows, though I'm not sure if this'll end up being much faster in the end:

import collections
import datetime
import pandas

ID_Use_Totals = collections.defaultdict(list)
ID_Order_Dates = collections.defaultdict(list)
ID_Received_Dates = collections.defaultdict(list)
ID_Refs = {}
IDs = set(args.ID)
order_actions = {"Order/Resupply", "Cons. Purchase"}

for filename in args.csv:
    for index, row in pd.read_csv(filename).iterrows():
        item = row["Stock Item"]
        if item not in IDs:
            continue
        ref = row["Ref"]
        action = row["Action"]
        if action in order_actions:
            date = datetime.datetime.fromisoformat(row["TransDate"])
            ID_Order_Dates[item].append({ref: date})
        elif action == "Received":
            date = datetime.datetime.fromisoformat(row["TransDate"])
            ID_Received_Dates[item].append({ref: date})
        elif action == "Use":
            ID_Use_Totals[item].append(row["Qty"])

Answer 2

You can use the apply function. The code will look like this:

df.apply(TSQs, axis=1)

Here when axis=1 , each row will be sent to the function TSQs as a pd.Series from where you can index like row["Ref"] to get value of that line. Since this is a vector operation, it will run so much after that a for loop.

Answer 3

Probably fastest not to iterate at all:

# Build some boolean indices for your various conditions
idx_stock_item = df["Stock Item"].isin(IDs)
idx_purchases =  df["Action"].isin(['Order/Resupply', 'Cons. Purchase'])
idx_order_dates = df["Stock Item"].isin(ID_Order_Dates)

# combine the indices to act on specific rows all at once
idx_combined = idx_stock_item & idx_purchases & ~idx_order_dates
# It looks like you were putting a single entry dictionary in each row - wouldn't it make sense to rather just use two columns? i.e. take advantage of the DataFrame data structure
ID_Order_Dates.loc[df.loc[idx_combined, "Stock Item"], "Ref"] = df.loc[idx_combined, "Ref"]   
ID_Order_Dates.loc[df.loc[idx_combined, "Stock Item"], "Date"] = df.loc[idx_combined, "TransDate"]

# repeat for your other cases
# ...