Pandas join two dataframes based on multiple dates

Question

I have two pandas DataFrames , where I am trying to preform an advanced join. In the following example I want to join df and df2 based on my_key where the date range from_dt and to_dt have an overlap. How can I do this using pandas?

Ex df:

value, my_key, from_dt, to_dt
1, a, 2007-01-01, 2009-02-01
2, b, 2001-01-01, 2011-01-01
3, c, 2015-01-01, 2020-01-01

df2:

my_key, value2, from_dt, to_dt
a, a1, 2007-01-01, 2008-01-01
a, a2, 2008-01-01, 2010-01-01
b, b1, 2009-01-01, 2015-01-01
c, c1, 2011-01-01, 2011-12-30

desired result:

value, value2, from_dt, to_dt
1, a1, 2007-01-01, 2008-01-01
1, a2, 2008-01-01, 2009-02-01
2, b1, 2009-01-01, 2011-01-01

Answer 1

@Jianxun's answer is great-- note also that if your data is in CSV as the question seems to suggest, you can get pd.datetime automatically with

df = pd.read_csv("df.csv", parse_dates=True)

You may want to check out these sections of the pandas docs.

Answer 2

This can be done in two steps. First do a outer-merge, and second keep the rows that do overlap.

import pandas as pd

# your data
# ===================================
df

   value my_key     from_dt       to_dt
0      1      a  2007-01-01  2009-02-01
1      2      b  2001-01-01  2011-01-01
2      3      c  2015-01-01  2020-01-01

df2

  my_key value2     from_dt       to_dt
0      a     a1  2007-01-01  2008-01-01
1      a     a2  2008-01-01  2010-01-01
2      b     b1  2009-01-01  2015-01-01
3      c     c1  2011-01-01  2011-12-30

# processing
# ======================================
# outer merge
df_temp = pd.merge(df, df2, on=['my_key'], how='outer')

# just make sure that the columns are in proper datetime type
# you don't have to do this if your data is already in datetime
df_temp.from_dt_x = pd.to_datetime(df_temp.from_dt_x)
df_temp.to_dt_x = pd.to_datetime(df_temp.to_dt_x)
df_temp.from_dt_y = pd.to_datetime(df_temp.from_dt_y)
df_temp.to_dt_y = pd.to_datetime(df_temp.to_dt_y)

   value my_key  from_dt_x    to_dt_x value2  from_dt_y    to_dt_y
0      1      a 2007-01-01 2009-02-01     a1 2007-01-01 2008-01-01
1      1      a 2007-01-01 2009-02-01     a2 2008-01-01 2010-01-01
2      2      b 2001-01-01 2011-01-01     b1 2009-01-01 2015-01-01
3      3      c 2015-01-01 2020-01-01     c1 2011-01-01 2011-12-30

# get rows that do overlap
result = df_temp[(df_temp.to_dt_x >= df_temp.from_dt_y) & (df_temp.from_dt_x <= df_temp.to_dt_y)]

   value my_key  from_dt_x    to_dt_x value2  from_dt_y    to_dt_y
0      1      a 2007-01-01 2009-02-01     a1 2007-01-01 2008-01-01
1      1      a 2007-01-01 2009-02-01     a2 2008-01-01 2010-01-01
2      2      b 2001-01-01 2011-01-01     b1 2009-01-01 2015-01-01

Update:

result['from_dt'] = result[['from_dt_x', 'from_dt_y']].max(axis=1)
result['to_dt'] = result[['to_dt_x', 'to_dt_y']].min(axis=1)
result.drop(['from_dt_x', 'to_dt_x', 'from_dt_y', 'to_dt_y'], axis=1)

   value my_key value2    from_dt      to_dt
0      1      a     a1 2007-01-01 2008-01-01
1      1      a     a2 2008-01-01 2009-02-01
2      2      b     b1 2009-01-01 2011-01-01