create a new column in pandas dataframe using if condition from another dataframe

Question

I have two dataframes as follows

transactions

    buy_date    buy_price
0   2018-04-16  33.23
1   2018-05-09  33.51
2   2018-07-03  32.74
3   2018-08-02  33.68
4   2019-04-03  33.58

and

cii

    from_fy     to_fy       score
0   2001-04-01  2002-03-31  100
1   2002-04-01  2003-03-31  105
2   2003-04-01  2004-03-31  109
3   2004-04-01  2005-03-31  113
4   2005-04-01  2006-03-31  117

In the transactions dataframe I need to create a new columns cii_score based on the following condition

if transactions['buy_date'] is between cii['from_fy'] and cii['to_fy'] take the cii['score'] value for transactions['cii_score']

I have tried list comprehension but it is no good.

Request your inputs to tackle this.

Answer 1

First, we set up your dfs. Note I modified the dates in transactions in this short example to make it more interesting

import pandas as pd
from io import StringIO
trans_data = StringIO(
    """
,buy_date,buy_price
0,2001-04-16,33.23
1,2001-05-09,33.51
2,2002-07-03,32.74
3,2003-08-02,33.68
4,2003-04-03,33.58
    """
)

cii_data = StringIO(
    """
,from_fy,to_fy,score
0,2001-04-01,2002-03-31,100
1,2002-04-01,2003-03-31,105
2,2003-04-01,2004-03-31,109
3,2004-04-01,2005-03-31,113
4,2005-04-01,2006-03-31,117    
    """
)
tr_df = pd.read_csv(trans_data, index_col = 0)
tr_df['buy_date'] = pd.to_datetime(tr_df['buy_date'])

cii_df = pd.read_csv(cii_data, index_col = 0)
cii_df['from_fy'] = pd.to_datetime(cii_df['from_fy'])
cii_df['to_fy'] = pd.to_datetime(cii_df['to_fy'])

The main thing is the following calculation: for each row index of tr_df find the index of the row in cii_df that satisfies the condition. The following calculates this match, each element of the list is equal to the appropriate row index of cii_df :

match = [ [(f<=d) & (d<=e) for f,e in zip(cii_df['from_fy'],cii_df['to_fy']) ].index(True) for d in tr_df['buy_date']]
match

produces

[0, 0, 1, 2, 2]

now we can merge on this

tr_df.merge(cii_df, left_on = np.array(match), right_index = True)

so that we get

    key_0 buy_date  buy_price   from_fy to_fy       score
0   0   2001-04-16  33.23   2001-04-01  2002-03-31  100
1   0   2001-05-09  33.51   2001-04-01  2002-03-31  100
2   1   2002-07-03  32.74   2002-04-01  2003-03-31  105
3   2   2003-08-02  33.68   2003-04-01  2004-03-31  109
4   2   2003-04-03  33.58   2003-04-01  2004-03-31  109

and the score column is what you asked for