How to aggregate columns of sets?

Question

I have a pandas datafame where the rows in a particular column are sets of id's. I would like to aggregate across a 15min period and find all such unique id's.

timestamp  |         ids           |  some_int
00:03:00     {id1, id2, id3}           5
00:10:00     {id2, id4, id7, id10}     9
00:25:00     {id7, id22, id24}         10
00:45:00     {id23, id30}              24


df.resample('15min').agg({'ids': ??, 'some_int': sum)

I've tried sum and a few other transformations on the ids column but I don't quite have it yet.

Answer 1

Change set to list then using sum

df.ids=df.ids.apply(list)
s=df.resample('15min').agg({'ids': 'sum', 'some_int': 'sum'})
s.loc[s.ids.eq(False),'ids']=''
s.ids=s.ids.apply(set)
s
Out[134]: 
                                                 ids  some_int
timestamp                                                     
2018-02-27 00:00:00  {id2, id4, id7, id10, id1, id3}        14
2018-02-27 00:15:00                {id24, id7, id22}        10
2018-02-27 00:30:00                               {}         0
2018-02-27 00:45:00                     {id23, id30}        24

Answer 2

Here is one way. For some reason I couldn't get agg working with set.union , so I performed 2 groupby operations and joined them.

import pandas as pd, numpy as np
from itertools import chain

df = pd.DataFrame([['00:03:00', {'id1', 'id2', 'id3'}, 5],
                   ['00:10:00', {'id2', 'id4', 'id7', 'id10'}, 9],
                   ['00:25:00', {'id7', 'id22', 'id24'}, 10],
                   ['00:45:00', {'id23', 'id30'}, 24]],
                  columns=['timestamp', 'ids', 'some_int'])

df['timestamp'] = pd.to_datetime(df['timestamp'])
df = df.set_index('timestamp')

x = df.resample('15min')['ids'].apply(chain.from_iterable).map(set).reset_index()
y = df.resample('15min')['some_int'].apply(sum).reset_index()

pd.merge(x, y, how='left')

#             timestamp                              ids  some_int
# 0 2018-02-27 00:00:00  {id1, id4, id2, id3, id10, id7}        14
# 1 2018-02-27 00:15:00                {id22, id7, id24}        10
# 2 2018-02-27 00:30:00                               {}         0
# 3 2018-02-27 00:45:00                     {id30, id23}        24

Answer 3

You can use set.union but not directly on the pandas Series. First, you have to unpack the series.

df.resample('15min').agg({'ids': lambda s: set.union(*s), 'some_int': sum)