如何有效地将函数应用于大型pandas数据帧的行？

Question

Im trying to create a training dataset for a model from a current data set. Its for blackjack and each row is how a player plays their hands.

The table might look something like this:

|Card1|Card2|Card3|Card4|Card5|PlayerTotal|DealerCard1|Win/Lose
|   7 | 10  |  0  |  0  |  0  |  17       |    10     |  0
|   4 | 3   |  10 |  0  |  0  |  17       |     8     |  1

Id like to turn it into rows with just the sum of the players hand, the dealers card and the win/lose. However, if more than 2 cards have been played (so the player hit) then id like to make multiple rows for that sample with the game at each stage (so before the player hits each time)

So the example would become:

|PlayerTotal|DealerCard1|Win/Lose
|    17     |     10    |  0
|    7      |     8     |  1
|    17     |     8     |  1

How can I do this efficiently?

I can do this fine with a small dataset using pd.apply and a custom function with if statements, but once I use the whole dataset (~1 mill points) its very slow and memory intensive.

Something like this:

def extractRounds(x):
    totals = []
    totals.append(x[0:2], x[5], x[6]])

    if x[2] > 0:
        totals.append([sum(x[0:3]), x[5], x[6]])
    else:
        return pd.Series(totals)

    if x[3] > 0:
        totals.append([sum(x[0:4]), x[5], x[6]])
    else:
        return pd.Series(totals)

    if x[4] > 0:
        totals.append([sum(x[0:5]), x[5], x[6]])

    return pd.Series(totals)


b = (a.apply(extractRounds, axis = 1)).stack()

Im guessing that it is the extractRounds(x) function that isn't the most effective or efficient.

So im wondering if I am barking up the wrong tree trying to do this by applying a function to each row or if there is a better way?

Let me know if this isn't clear. Cheers!

Answer 1

You can use melt to convert your data into long format, add a cumulative sum, and then just exclude the zero card values for cards 3-5. And exclude card 1 since the player will always have a minimum of 2 cards.

Here's your example as a dataframe:

import pandas as pd
import numpy as np

raw = pd.DataFrame({'Card1': [7, 4],
                    'Card2': [10, 3],
                    'Card3': [0, 10],
                    'Card4': [0, 0],
                    'Card5': [0, 0],
                    'DealerCard1': [10, 8],
                    'PlayerTotal': [17, 17],
                    'Win/Lose': [0, 1]})

raw.index.name = 'Game'

Use melt to create another dataframe in long format:

df = (raw.reset_index()
     .melt(value_vars=['Card1', 'Card2', 'Card3', 'Card4', 'Card5'], 
           id_vars=['Game', 'DealerCard1', 'Win/Lose'],
           value_name='CardValue', 
           var_name='Card')
     .sort_values('Game')
     .reset_index(drop=True))

Recreate the PlayerTotal column as a cumulative sum:

df['PlayerTotal'] = df.groupby('Game')['CardValue'].apply(np.cumsum)

And then you can just exclude card 1 and the zero cards and select your desired columns:

df.loc[(df['CardValue']!=0) & (df['Card']!='Card1'), ['PlayerTotal', 'DealerCard1', 'Win/Lose']]

That will give you:

PlayerTotal DealerCard1 Win/Lose
1   17  10  0
6   7   8   1
7   17  8   1

Answer 2

You can use command-line tools to add the extra lines to the csv file and do the summation.

Let's say first few lines of CSV file data.csv is

Card1,Card2,Card3,Card4,Card5,PlayerTotal,DealerCard1,Win/Lose
7,10,0,0,0,17,10,0
4,3,10,0,0,17,8,1

Running the following command gives us the desired output

sed 's/\(.*,\)\(.*,\)\([1-9][0-9]*,\)\(.*,.*,.*,.*,.*\)/\1\2\3\4\n\1\20,\4/' data.csv | cut -d ',' -f 1,2,3,7,8 | awk -F ',' 'NR>1 {print $1+$2+$3 "," $4 "," $5}' > data_2.csv

It creates a file named data_2.csv containing

17,10,0
17,8,1
7,8,1

--------------------------------

Explanation of the command:

sed 's/\(.*,\)\(.*,\)\([1-9][0-9]*,\)\(.*,.*,.*,.*,.*\)/\1\2\3\4\n\1\20,\4/' data.csv

reads the data.csv line by line, if a line has 0 value in third column, it adds another line where the third column is 0.

| cut -d ',' -f 1,2,3,7,8

reads the data from previous step and filter the data to columns 1,2,3,7,8 (these are the columns we care about)

| awk -F ',' 'NR>1 {print $1+$2+$3 "," $4 "," $5}' > data_2.csv

reads the data from previous step, adds up the first three columns and writes it in a file called data_2.csv together with the last two columns.