如何根据 python 中的多个标准连接两个数据帧？

Question

I have the following two dataframes:

Shortlist:

ticker     date     open    high    low     close   volume
ABC     2000-12-29  0.450   0.455   0.445   0.455   205843.0
ABC     2001-01-31  0.410   0.410   0.405   0.410   381500.0
ABC     2001-02-28  0.380   0.405   0.380   0.400   318384.0
...
ABC     2001-06-30  0.430   0.445   0.430   0.440   104016.0

MCap

Code    EOM       mcRank    MktCap
ABC    29/12/2000   74     1563.967892
ABC    31/03/2001   98     998.156279
ABC    30/06/2001   59     2035.603350

I now want to create a new table that adds the columns of mcRank and MktCap from the MCap dataframe to the Shortlist dataframe, where the Code and the Date match. If the date is shortlist is between the dates in MCap it should use the last known date.

The result should like this:

ticker     date     open    high    low     close   volume    mcRank    MktCap
ABC     2000-12-29  0.450   0.455   0.445   0.455   205843.0   74     1563.967892
ABC     2001-01-31  0.410   0.410   0.405   0.410   381500.0   74     1563.967892   
ABC     2001-02-28  0.380   0.405   0.380   0.400   318384.0   74     1563.967892
...
ABC     2001-06-30  0.430   0.445   0.430   0.440   104016.0   59     2035.603350

I've tried pd.concat and pd.merge - but can't seem to get the right results.

Answer 1

What you want to do is

First align both date format, you can process it as string which makes it easier

Second pd.merge them, use left_on, right_on and how='outer' to merge everything, and PURPOSELY create NA values

Then you can use DataFrame.fillna(method='ffill') to fill na base on previous values

Answer 2

Well this seems like a merge task, but first make sure EOM and date columns are actually the same variable dtype ( datetime ).

shortlist['date'] = pd.to_datetime(shortlist['date'], format='%Y-%m-%d')
MCap['EOM'] = pd.to_datetime(MCap['EOM'], format='%d/%m/%Y')

And then do the merge (this wont work if ticker or Codes are the indexes, if they are, reset the index first ie shortlist.rest_index(inplace=True) ):

new_df = shortlist.merge(how='left', left_on=['ticker', 'date'], right_on=['Code', 'EOM']).reset_index()

Answer 3

IIUC,

you might have to break down the steps: first merge the two dataframes (i use the join function) on the dates, then fill in the null values with the oldest date from mcap (I am using the result of ur output as a guide):

convert to datetime and set index

df['date'] = pd.to_datetime(df['date'], format = '%Y-%m-%d')
df = df.set_index('date')
mcap['EOM'] = pd.to_datetime(mcap['EOM'])
mcap = mcap.set_index("EOM")

combine the dataframes:

res = df.join(mcap)

get the indices for the null rows:

indices = res[res.isna().any(axis=1)].index

get the values from mcap for the oldest date:

latest_mcap = mcap.loc[mcap.index.min()].tolist()

assign latest_mcap to the null values in res:

res.loc[indices,['Code','mcRank','MktCap']] = latest_mcap

ticker  open    high    low close   volume  Code    mcRank  MktCap
date                                    
2000-12-29  ABC 0.45    0.455   0.445   0.455   205843.0    ABC 74.0    1563.967892
2001-01-31  ABC 0.41    0.410   0.405   0.410   381500.0    ABC 74.0    1563.967892
2001-02-28  ABC 0.38    0.405   0.380   0.400   318384.0    ABC 74.0    1563.967892
2001-06-30  ABC 0.43    0.445   0.430   0.440   104016.0    ABC 59.0    2035.603350