Python - 用基于另一个数据帧中的索引的条件填充数据帧的最快方法

Question

I have data in an input dataframe (input_df). Based on an index in another benchmark dataframe (bm_df), I would like to create a third dataframe (output_df) that is populated based on a condition using the indices in the original two dataframes.

For each date in the index for the bm_df I would like to populate my output using the latest data available in the input_df, subject to the condition that the data has an index date before or equal to that in the bm_df. For example, in the case study data below the output dataframe for the first index date (2019-01-21) would be populated with the data from the input_df datapoint for the 2019-01-21. However, if a datapoint for the 2019-01-21 did not exist this would use the 2019-01-18.

The use case here is mapping and backfilling large datasets for the latest data available for a given date. I have written up some python to do this for me (which works), however I think there is probably a more pythonic and therefore faster way to implement the solution. My underlying dataset this is applied to has large dimensions in terms of the number of columns and length of the columns and so I would like something as efficient as possible - my current solution is too slow when run on the full dataset I am using.

Any help is much appreciated!

input_df:

index   data
2019-01-21  0.008
2019-01-18  0.016
2019-01-17  0.006
2019-01-16  0.01
2019-01-15  0.013
2019-01-14  0.017
2019-01-11  0.017
2019-01-10  0.024
2019-01-09  0.032
2019-01-08  0.012

bm_df:

index   
2019-01-21  
2019-01-14  
2019-01-07  

output_df:

index   data
2019-01-21  0.008
2019-01-14  0.017
2019-01-07  NaN

Please see the code I am currently using below:

import pandas as pd
import numpy as np

# Import datasets
test_index = ['2019-01-21','2019-01-18','2019-01-17','2019-01-16','2019-01-15','2019-01-14','2019-01-11','2019-01-10','2019-01-09','2019-01-08']    
test_data = [0.008, 0.016,0.006,0.01,0.013,0.017,0.017,0.024,0.032,0.012]
input_df= pd.DataFrame(test_data,columns=['data'], index=test_index)

test_index_2= ['2019-01-21','2019-01-14','2019-01-07']  
bm_df= pd.DataFrame(index=test_index_2)

#Preallocate
data_mat= np.zeros([len(bm_df)])

#Loop over bm_df index and find the most recent variable from input_df which from a date before the index date 
for i in range(len(bm_df)):
    #First check to see if there are no dates before the selected date, if true fill with NaN
    if sum(input_df.index <= bm_df.index[i])>0:
        data_mat[i] = input_df['data'][max(input_df.index[input_df.index <= bm_df.index[i]])]
    else:
        data_mat[i] = float('NaN')

output_df= pd.DataFrame(data_mat,columns=['data'],index=bm_df.index)

Answer 1

I have not tested the execution time, but I would rely on join being referenced as efficient in pandas documentation :

... Efficiently join multiple DataFrame objects by index at once...

And I would use shift to get the value for the highest date before the searched one.

All that give:

output_df = bm_df.join(input_df.shift(-1), how='left')

             data
2019-01-21  0.016
2019-01-14  0.017
2019-01-07    NaN

---

This approach is indeed far less versatile that explicit loops. It is the price for pandas vectorization. For example for a less than or equal to condition the code will be slightly different. Here is an example with an additional date in bm_df not present in input_df :

...
test_index_2= ['2019-01-21','2019-01-14','2019-01-13','2019-01-07']  
...
tmp_df = input_df.join(bm_df).fillna(method='bfill')
output_df = bm_df.join(tmp_df, how='inner')

And we obtain as expected:

             data
2019-01-21  0.008
2019-01-14  0.017
2019-01-13  0.017
2019-01-07  0.012