Merging DFs from two different lists in python

Question

There are two lists where elements are DFs and having datetimeindex :

lst_1 = [ df1, df2, df3, df4]   #columns are same here 'price' 

lst_2 = [df1, df2, df3, df4]    #columns are same here 'quantity'

I am doing it with one by one using the pandas merge function. I tried to do something where i add the two list and make function like this:

def df_merge(df1 ,df1):
    p_q_df1 = pd.merge(df1,df1,  on='Dates') 
    return p_q_df1        

#this merged df has now price and quantity representing df1 from list! and list_2

still i have to apply to every pair again. Is there a better way, maybe in loop to automate this?

Answer 1

Consider elementwise looping with zip which can be handled in a list comprehension.

# DATES AS INDEX
final_lst = [pd.concat(i, j, axis=1) for i, j in zip(lst_1, lst_2)]

# DATES AS COLUMN
final_lst = [pd.merge(i, j, on='Dates') for i, j in zip(lst_1, lst_2)]

Answer 2

IIUC,

you could concat your df's then merge

dfs_1 = pd.concat(lst_1)
dfs_2 = pd.concat(lst_2)
pd.merge(dfs_1,dfs_2,on='Dates',how='outer') 
# change how to specify the behavior of the merge.

I'm assuming your dataframes are the same shape so they can be concatenated.

if you want to merge multiple dataframes in your list you can use the reduce function from the standard python lib using an outer merge to get every possible row.

from functools import reduce
lst_1 = [ df1, df2, df3, df4] 

df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['Dates'],
                                            how='outer'), lst_1)

Answer 3

lst_1 = [ df1, df2, df3, df4]   #columns are same here 'price' 

lst_2 = [df1, df2, df3, df4]    #columns are same here 'quantity'

def merge(lst_1, lst_2):
   df = pd.DataFrame()
   for _df in lst_1:
      df = df.merge(_df, on='Dates')

   for _df in lst_2:
      df = df.merge(_df, on='Dates')