用另一列的最新值填充数据框列

Question

I have two dataframes list1 and list2 that each have a different amount of rows with random indices. list1 has ~240,000 rows while list2 has ~390,000 rows. They are sorted from the earliest time to the latest according to the ['time'] column. They look roughly like this:

list1

     time    rates
299  09:31   1.30
1230 10:34   2.42
32   13:40   1.49
     ...   ...

list2

     time    Symbol    IV
78   10:31   aqb       7
121  10:59   cdd       3
3240 11:19   oty       4
393  13:54   zqb       8
44   14:13   omu       1
     ... 

Each row on list2 has a ['time'] value. I want each row in list2 to have the latest ['rates'] value from list1 that is no later than its own ['time'] value. Until then, the same ['rates'] value can be filled into list2 (sorry I know this is confusing). An example of the desired result with an explanation is shown below.

Desired result

     time    Symbol    IV    rates
78   10:31   aqb       7     1.30
121  10:59   cdd       3     2.42
3240 11:19   oty       4     2.42
393  13:54   zqb       8     1.49
44   14:13   omu       1     1.49

The first row in list1 is from 9:31, and the second row is from 10:34. The first row in list2 is at 10:31, so it should be filled with the ['rates'] value from 9:31 instead of the rates value from 10:34, since 10:34 is later than 10:31. Next row in list2 is 10:59. The latest row in list1 that is not after 10:59 is 10:34, so the value 2.42 from 10:34 is filled in. The same the third row in list2 with 11:19.

How do I go about accomplishing this without using a for loop to slowly iterrows() through every single row and doing a bunch of the above if else checks that would take an eternity given the few hundred thousand rows in each dataframe? Thanks!

Answer 1

Using merge_asof

df1.time=pd.to_datetime(df1.time,format='%H:%M')
df2.time=pd.to_datetime(df2.time,format='%H:%M')
pd.merge_asof(df2.sort_values('time'),df1.sort_values('time'),on='time',direction = 'backward' )
Out[79]: 
                 time Symbol  IV  rates
0 1900-01-01 10:31:00    aqb   7   1.30
1 1900-01-01 10:59:00    cdd   3   2.42
2 1900-01-01 11:19:00    oty   4   2.42
3 1900-01-01 13:54:00    zqb   8   1.49
4 1900-01-01 14:13:00    omu   1   1.49

Answer 2

I simply merged the two dataframes on ['time'] with an indicator then sorted the new dataframe on ['time']:

list2 = list2.merge(list1,how = 'outer', on= ['time'], indicator = True)
list2 = list2.sort_values(['time'])

and then filled rows with 'left_only' indicator that consequently have Nan ['rates'] values with the latest values from rows with an 'right_only' indicator by using:

list2= list2.fillna(method = 'ffill')

Then dropped the rows from list1 with:

list2= list2.loc[list2['_merge']!= 'right_only']