Python：在 Pandas GroupBy 对象上有效地使用 apply

Question

I am trying to perform a task which is conceptually simple, but my code seems to be way too expensive. I am looking for a faster way, potentially utilizing pandas' built-in functions for GroupBy objects.

The starting point is a DataFrame called prices, with columns=[['item', 'store', 'day', 'price']], in which each observatoin is the most recent price update specific to a item-store combination. The problem is that some price updates are the same as the last price update for the same item-store combination. For example, let us look at a particular piece:

       day  item_id  store_id  price
35083   34    85376       211   5.95
56157   41    85376       211   6.00
63628   50    85376       211   5.95
64955   51    85376       211   6.00
66386   56    85376       211   6.00
69477   69    85376       211   5.95

In this example I would like the observation where day equals 56 to be dropped (because price is the same as the last observation in this group). My code is:

def removeSameLast(df):

    shp = df.shape[0]
    lead = df['price'][1:shp]
    lag = df['price'][:shp-1]
    diff = np.array(lead != lag)

    boo = np.array(1)
    boo = np.append(boo,diff)
    boo = boo.astype(bool)

    df = df.loc[boo]

    return df

gCell = prices.groupby(['item_id', 'store_id'])
prices = gCell.apply(removeSameLast)

This does the job, but is ugly and slow. Sorry for being a noob, but I assume that this can be done much faster. Could someone please propose a solution? Many thanks in advance.

Answer 1

I would suggest going for a simple solution using the shift function from Pandas. This would remove the use of the groupby and your function call.

The idea is to see where the Series [5.95, 6, 5.95, 6, 6, 5.95] is equal to the shifted one, [nan, 5.95, 6, 5.95, 6, 6] and delete(or just don't select) the rows where this condition happens.

>>> mask = ~np.isclose(prices['price'], prices['price'].shift())
>>> prices[mask]
       day  item_id store_id    price
35083   34    85376      211    5.95
56157   41    85376      211    6.00
63628   50    85376      211    5.95
64955   51    85376      211    6.00
69477   69    85376      211    5.95

Simple benchmark:

%timeit prices = gCell.apply(removeSameLast)
100 loops, best of 3: 4.46 ms per loop

%timeit mask = df.price != df.price.shift()
1000 loops, best of 3: 183 µs per loop