Python Pandas Dataframe: based on DateTime criteria, I would like to populate a dataframe with data from another dataframe

Question

I have created a simple dataframe,"F_test". I would now like to populate another dataframe,"P", with data from "F_test" based on whether the cell in "P" lies in the same row as "F_test" and is inbetween the startdates/enddates for that row.

However, when I execute a simple For Loop to do this, after the first row, no other data is updated in the "P" matrix.

In the code on my PC, I actually extract "F_test" data from an Excel File, but for the purposes of giving a complete dataset on this forum, I have manually created the simple dataframe, named "F_test".

As you may be able to tell from the code, I am a recent convert from the Matlab/VBA Excel world...

I would really appreciate your wisdom on this topic.

F0 = ('08/02/2018','08/02/2018',50)
F1 = ('08/02/2018','09/02/2018',52)
F2 = ('10/02/2018','11/02/2018',46)
F3 = ('12/02/2018','16/02/2018',55)
F4 = ('09/02/2018','28/02/2018',48)
F_mat = [[F0,F1,F2,F3,F4]]
F_test = pd.DataFrame(np.array(F_mat).reshape(5,3),columns= ('startdate','enddate','price'))

#convert string dates into DateTime data type
F_test['startdate'] = pd.to_datetime(F_test['startdate'])
F_test['enddate'] = pd.to_datetime(F_test['enddate'])

#convert datetype to be datetime type for columns startdate and enddate
F['startdate'] = pd.to_datetime(F['startdate'])
F['enddate'] = pd.to_datetime(F['enddate'])

#create contract duration column
F['duration'] = (F['enddate'] - F['startdate']).dt.days + 1

#re-order the F matrix by column 'duration', ensure that the bootstrapping 
#prioritises the shorter term contracts 
F.sort_values(by=['duration'], ascending=[True])

#create D matrix, dataframe containing each day from start to end date
tempDateRange = pd.date_range(start=F['startdate'].min(), end=F['enddate'].max(), freq='D')
D = pd.DataFrame(tempDateRange)

#define Nb of Calendar Days in a variable to be used later
intNbCalendarDays = (F['enddate'].max() - F['startdate'].min()).days + 1

#define Nb of Contracts in a variable to be used later
intNbContracts = len(F)

#define a zero filled matrix, P, which will house the contract prices 
P = pd.DataFrame(np.zeros((intNbContracts, intNbCalendarDays)))

#rename columns of P to be the dates contained in matrix array D
P.columns = tempDateRange 

#create prices in correct rows in P
for i in list(range(0, intNbContracts)):
    for j in list(range(0, intNbCalendarDays)):
        if ((F.iloc[i,0] >= P.columns[j]) & (F.iloc[i,1] <= P.columns[j] )):
            P.iloc[i,j] = F.iloc[i,2]
P

Answer 1

I think your date comparisons are the wrong way round at the end and you should use 'and' not '&' (which is the bitwise operator). Try this:

# create prices in correct rows in P
for i in list(range(0, intNbContracts)):
    for j in list(range(0, intNbCalendarDays)):
        if (F.iloc[i, 0] <= P.columns[j]) and (F.iloc[i, 1] >= P.columns[j]):
            P.iloc[i, j] = F.iloc[i, 2]

Answer 2

This is still probably not as efficient as you could get, but is better I think. This would replace from your "#create D matrix, dataframe containing..." comment onwards

# create prices P
P = pd.DataFrame()
for index, row in F.iterrows():
    new_P_row = pd.Series()
    for date in pd.date_range(row['startdate'], row['enddate']):
        new_P_row[date] = row['price']
    P = P.append(new_P_row, ignore_index=True)

P.fillna(0, inplace=True)