如何根據每個 dataframe 中兩個不同日期列的日期條件合並兩個數據框？

Question

我有兩個數據框的形式：

數據框（df1）：

P_CLIENT_ID	P_DATE_ENCOUNTER
25835	2016-12-21
25835	2017-02-21
25835	2017-04-25
25835	2017-06-21
25835	2017-09-04
25835	2018-01-08
25835	2018-04-03

數據框（df2）：

R_CLIENT_ID	R_DATE_TESTED	R_RESULT
25835	2017-03-07	20.0
25835	2017-08-03	20.0
25835	2018-03-23	20.0
25835	2019-06-28	20.0
25835	2019-08-19	42.0
25835	2020-04-20	40.0
25835	2021-06-03	20.0

我想將 df2 合並到 df1（主表）上，連接鍵為P_CLIENT_ID和R_CLIENT_ID ，附加最新的R_DATE_TESTED和R_RESULT

第一個條件：如果R_DATE_TESTED > P_DATE_ENCOUNTER則使R_DATE_TESTED, R_RESULT字段無效。

第二個條件：如果R_DATE_TESTED < P_DATE_ENCOUNTER然后將最近的R_DATE_TESTED, R_RESULT字段附加到 dataframe，最終結果為：

邏輯結果應如下所示：

P_CLIENT_ID	R_CLIENT_ID	P_DATE_ENCOUNTER	R_DATE_TESTED	R_RESULT
25835	25835.0	2016-12-21	鈉鹽	鈉鹽
25835	25835.0	2017-02-21	鈉鹽	鈉鹽
25835	25835.0	2017-04-25	2017-03-07	20.0
25835	25835.0	2017-06-21	2017-03-07	20.0
25835	25835.0	2017-09-04	2017-08-03	20.0
25835	25835.0	2018-01-08	2017-08-03	20.0
25835	25835.0	2018-04-03	2018-03-23	20.0

注意：實際數據集相當大：df1 ~ 700000 行和 df2 ~ 125000 行

代碼嘗試

import pandas as pd
import numpy as np

df1 = pd.DataFrame({'P_CLIENT_D': ['25835','25835','25835','25835','25835','25835','25835'],
                    'P_DATE_ENCOUNTER': ['2016-12-21','2017-02-21','2017-04-25','2017-06-21','2017-09-04','2018-01-08','2018-04-03']})

df2 = pd.DataFrame({'R_CLIENT_ID': ['25835','25835','25835','25835','25835','25835','25835'],
                    'R_DATE_TESTED': ['2017-03-07','2017-08-03','2018-03-23','2019-06-28','2019-08-19','2020-04-20','2021-06-03'],
                   'R_RESULT':[20,20,20,20,42,40,20]})

df_merged = pd.merge(df1, df2, left_on=['P_CLIENT_D'], right_on = ['R_CLIENT_ID'],  how='left')

df_merged = df_merged.drop_duplicates(subset=['P_CLIENT_D', 'P_DATE_ENCOUNTER'], keep='last')

df_merged['FLAG_LAB_AFTER_VISIT'] = 0
df_merged.loc[df_merged.R_DATE_TESTED >= df_merged.P_DATE_ENCOUNTER,'FLAG_LAB_AFTER_VISIT']=1
print(df_merged['FLAG_LAB_AFTER_VISIT'].sum(), 'future labs set to null')

#now the rows with flags - set all lab fields to null
df_merged.loc[df_merged['FLAG_LAB_AFTER_VISIT']==1, df2.columns] = np.nan

Answer 1

嘗試使用pandas.merge_asof ：

>>> pd.merge_asof(df1, 
                  df2, 
                  left_on="P_DATE_ENCOUNTER", 
                  right_on="R_DATE_TESTED", 
                  left_by="P_CLIENT_ID", 
                  right_by="R_CLIENT_ID")

   P_CLIENT_ID P_DATE_ENCOUNTER  R_CLIENT_ID R_DATE_TESTED  R_RESULT
0        25835       2016-12-21          NaN           NaT       NaN
1        25835       2017-02-21          NaN           NaT       NaN
2        25835       2017-04-25      25835.0    2017-03-07      20.0
3        25835       2017-06-21      25835.0    2017-03-07      20.0
4        25835       2017-09-04      25835.0    2017-08-03      20.0
5        25835       2018-01-08      25835.0    2017-08-03      20.0
6        25835       2018-04-03      25835.0    2018-03-23      20.0