Group timestamps in one dataframe by ranges in another dataframe

Question

I have two dataframes with timestamps in different intervals and values, and I want to group the values in the first dataframe by the ranges in the second dataframe.

My first dataframe looks like this:

My second dataframe looks like this:

For example, I want to group all the values in the first dataframe whose timestamp falls between the first two timestamps in the second dataframe with the group_id in the second dataframe like this

I have a working code on small samples of these two dataframes like this:

sequence = list()
for i in range(len(df2)-1):
  print(list(df2['timestamp'])[i])
  seq_list = []
  while j < len(df1):
    if list(df1['timestamp'])[j] >= list(df2['timestamp'])[i+1]:
      break
    if list(df1['timestamp'])[j] >= list(df2['timestamp'])[i] and list(df1['timestamp'])[j] < list(df2['timestamp'])[i+1]:
      seq_list.append(list(df1['value'])[j])
    j += 1
  sequence.append([seq_list, list(df2['group_id'])[i]])```

But it is taking a long time to run on full dataframes, how can I optimize to get it run in reasonable times? 

Answer 1

Since df2 is all 10 min intervals, we can round down the times in df1 to match the times in df2. Since now df1 and df2 have the same times, we can create a map to the groups in df2 to groupby with.

This is a dict of your df2 timestamp to group_id

groupMap = df2.set_index('timestamp')['group_id'].to_dict()

Next you can do your group by statement with intervals of 10 min by rounding down, and map it to the groups from groupMap.

df1.groupby(df1['timestamp'].dt.floor('10 min').map(groupMap))

Then you can do whatever function you want to do on the groups.

Edit: If you want to see the group_id on df1, you can do the following:

df1['group_id'] = df1['timestamp'].dt.floor('10 min').map(groupMap)