创建库存历史记录的最有效方法是什么

Question

I am trying to get, for each day of the year, how many cars I had in stock and for how many days each car was in stock in that date.

I have the full history of movements (the timestamp each car was moved in and out of stock - for rent, for sale, repair and so on). Like this:

car             in          out        status_id    operation
PZR4010 08/02/2018 08:55    08/02/2018 16:29    12  out_stock
QRX0502 07/02/2018 09:00    07/02/2018 10:28    7   in_stock
PYR8269 06/02/2018 17:10    09/02/2018 21:22    12  in_stock
QRG6455 06/02/2018 12:39                        8   sold
QRU1867 08/02/2018 08:00    09/02/2018 11:07    12  in_stock
PZR8528 06/02/2018 17:51    07/02/2018 07:46    10  out_stock
PZR7184 06/02/2018 16:00    08/02/2018 12:10    7   in_stock
PZR0386 08/02/2018 09:02    14/02/2018 14:53    10  out_stock
PZR8600 06/02/2018 16:00    07/02/2018 07:34    7   in_stock
PZR1787 06/02/2018 17:02    20/02/2018 17:33    12  in_stock

So, for each car, I have to join the whole consecutive time it has been in-stock, to know for how long it was in that state.

So for instance:

car     in                 out          status_id   operation
QRX0502 08/02/2018 08:55    09/02/2018 16:29    7   in_stock
QRX0502 07/02/2018 09:00    08/02/2018 08:55    7   in_stock
QRX0502 06/02/2018 17:10    07/02/2018 09:00    7   in_stock

Will become simply:

car          in                 out            status_id    operation
QRX0502 06/02/2018 17:10    09/02/2018 16:29    7   in_stock

Capturing the min timestamp in the 'in' column and the max timestamp in the 'out' column.

I have tried to use groupby + shift:

#'mov' is the dataframe with all the stock movements
# I create a columns to better filter on the groupby

mov['aux']=mov['car']+" - "+mov['operation']

#creating the base dataframe to be the output

hist_mov=pd.DataFrame(columns=list(mov.columns))

for line, operation in mov.groupby(mov['aux'].ne(mov['aux'].shift()).cumsum()):
    g_temp=operation.groupby(['car','operation',
        'aux']).agg({'in':'min','out':'max'}).reset_index()
    hist_mov=hist_mov.append(g_temp,sort=True)

The problem is that the whole database takes about 16 hours to run, and I will have to run it every day, to update inventory status.

I want to build something like:

Every new row added to the history will check if it is consecutive to any one in my new base (hist_mov). If so, update that line. If not, add as a new line.

Any ideas? Thanks!

Answer 1

I think something like this might be what you are after:

cols = ["car", "operation"]
pd.merge(df.groupby(cols)["in"].min().reset_index(), 
         df.groupby(cols)["out"].max().reset_index(), on=cols, how="outer")

Edit:

Hopefully this alleviates the problem outlined in the comments, using a trans_id column to recognise separate instances of a car coming back in and out:

df['trans_id'] = df['operation'].ne(df['operation'].shift()).astype(int) + df.index
cols = ["car", "trans_id", "operation"]
df_grouped = pd.merge(df.groupby(cols)["in"].min().reset_index(), 
         df.groupby(cols)["out"].max().reset_index(), on=cols, how="outer")
df_grouped.drop('trans_id', axis=1, inplace=True)
df_grouped

Answer 2

I have found the answer!

The code that I had first posted was almost right,but it had an unecessary loop.

1- First I sort the items by car and data of status change:

    mov=mov.sort_values(['car','in'],ascending=False)

2- Then I clusterzire by car and operation:

    mov['aux']=mov['car']+" - "+mov['operation']
    mov['cluster']=(mov.aux != mov.aux.shift()).cumsum()

3- Finally I can just group by thist cluster Id, and get the min "in" value and the max "out" value:

    hist_mov=mov.groupby(['cluster','car','operation']).agg({'in':'min',
          'out':'max'}).reset_index().copy()