根据熊猫中另一列中相似值的分组来创建新列

Question

Hi I have an event data frame with datetimes and event ids and sensor ids. I would like to group events that happen within one hour per sensor and if possible tag them with the group count. Original Data Frame

         sensor_id  event_id   time  
    0    A         e1            2017-02-14 05:30:00      
    1    A         e2            2017-02-14 05:45:00 
    2    A         e3            2017-02-14 08:30:00 
    3    B         e3            2017-02-14 05:20:00 
    4    B         e4            2017-02-14 05:30:00 
    5    B         e6            2017-02-14 05:45:00 
    6    C         e1            2017-02-14 05:30:00 
    7    C         e3            2017-02-14 07:30:00 
    8    C         e7            2017-02-14 09:35:00 

Desired Result:

         sensor_id  event_id      time                  group 
    0    A         e1            2017-02-14 05:30:00      1
    1    A         e2            2017-02-14 05:45:00      1
    2    A         e3            2017-02-14 08:30:00      2
    3    B         e3            2017-02-14 05:20:00      1
    4    B         e4            2017-02-14 05:30:00      1
    5    B         e6            2017-02-14 05:45:00      1
    6    C         e1            2017-02-14 05:30:00      1
    7    C         e3            2017-02-14 07:30:00      2
    8    C         e7            2017-02-14 09:35:00      3

I understand that I should group by user, event and then, time using timdelta of 1 hour but I have no clue how to do the rest. Any tips will be appreciated.

Answer 1

I think you need to go for dual groupby (Hope sensor_id is sorted if not we need to sorted them first) ie

df['new'] = df.groupby('sensor_id').apply( lambda x : x.groupby(x['time'].dt.hour).ngroup()+1).values

Output :

sensor_id event_id                time  new
0         A       e1 2017-02-14 05:30:00    1
1         A       e2 2017-02-14 05:45:00    1
2         A       e3 2017-02-14 08:30:00    2
3         B       e3 2017-02-14 05:20:00    1
4         B       e4 2017-02-14 05:30:00    1
5         B       e6 2017-02-14 05:45:00    1
6         C       e1 2017-02-14 05:30:00    1
7         C       e3 2017-02-14 07:30:00    2
8         C       e7 2017-02-14 09:35:00    3

Answer 2

You can use the pd.TimeGrouper + ngroup to group by time frequency.

df['time'] = pd.to_datetime(df.time)
df['group'] = df.set_index('time').groupby(['sensor_id', 
                    pd.TimeGrouper(freq='1H')], sort=False).ngroup().values

So far, we have what we want, but we'll need to reset the group value for each sensor_id , so another groupby call is in order.

df['group'] = df.groupby('sensor_id').group.apply(lambda x: x - x.min() + 1)

df

  sensor_id event_id                time  group
0         A       e1 2017-02-14 05:30:00      1
1         A       e2 2017-02-14 05:45:00      1
2         A       e3 2017-02-14 08:30:00      2
3         B       e3 2017-02-14 05:20:00      1
4         B       e4 2017-02-14 05:30:00      1
5         B       e6 2017-02-14 05:45:00      1
6         C       e1 2017-02-14 05:30:00      1
7         C       e3 2017-02-14 07:30:00      2
8         C       e7 2017-02-14 09:35:00      3