Python Pandas - 通过匹配主标识符将多个数据帧中的数据附加到同一行，如果没有来自该数据帧的结果则留空

Question

Very new to python and using pandas, I only use it every once in a while when I'm trying to learn and automate otherwise a tedious Excel task. I've come upon a problem where I haven't exactly been able to find what I'm looking for through Google or here on Stack Overflow.

I currently have 6 different excel (.xlsx) files that I am able to parse and read into data frames. However, whenever I try to append them together, they're simply added on as new rows in a final output excel files, but instead I'm trying to append similar data values onto the same row, and not the same column so that I can see whether or not this unique value shows up in these data sets or not. A shortened example is as follows

[df1]
0    Col1    Col2    
1    XYZ     41235
2    OAIS    15123
3    ABC     48938

[df2]
 0   Col1    Col2
 1   KFJ     21493
 2   XYZ     43782
 3   SHIZ    31299
 4   ABC     33347

[Expected Output]
 0    Col1    [df1]     [df2]    
 1    XYZ     41235     43782
 2    OAIS    15123     
 3    ABC     48938     33347
 4    KFJ               21493
 5    SHIZ              31299

I've tried to use a merge, however the actual data sheets are much more complicated in that I want to append 23 columns of data associated with each unique identifier in each data set. Such as, [XYZ] in [df2] has associated information across the next 23 columns that I would want to append after the 23 columns from the [XYZ] values in [df1].

How should I go about that? There are approximately 200 rows in each excel sheet and I would only need to essentially loop through until a matching unique identifier was found in [df2] with [df1], and then [df3] with [df1] and so on until [df6] and append those columns onto a new dataframe which would eventually be output as a new excel file.

df1 = pd.read_excel("set1.xlsx")
df2 = pd.read_excel("set2.xlsx")
df3 = pd.read_excel("set3.xlsx")
df4 = pd.read_excel("set4.xlsx")
df5 = pd.read_excel("set5.xlsx")
df6 = pd.read_excel("set6.xlsx")

Is currently the way I am reading into the excel files into data frames, I'm sure I could loop it, however, I am unsure of the best practices in doing so instead of hard coding each initialization of the data frame.

Answer 1

You can use the merge function.

pd.merge(df1, df2, on=['Col1'])

You can use multiple keys by adding to the list on .

You can read more about the merge function in here

If you need only certain of the columns you can reach it by:

df1.merge(df2['col1','col2']], on=['Col1'])

EDIT:

In case of looping through some df's you can loop through all df's except the first and merge them all:

df_list = [df2, df3, df4]

for df in df_list:
     df1 = df1.merge(df['col1','col2']], on=['Col1'])

Answer 2

You need merge with the parameter how = 'outer'

new_df = df1.merge(df2, on = 'Col1',how = 'outer', suffixes=('_df1', '_df2'))

You get

    Col1    Col2_df1    Col2_df2
0   XYZ     41235.0     43782.0
1   OAIS    15123.0     NaN
2   ABC     48938.0     33347.0
3   KFJ     NaN         21493.0
4   SHIZ    NaN         31299.0

Answer 3

For iterative merging, consider storing data frames in a list and then run the chain merge with reduce() . Below creates a list of dataframes from a list comprehension through the Excel files where enumerate() is used to rename the Col2 successively as df1 , df2 , etc.

from functools import reduce
...

dfList = [pd.read_excel(xl).rename(columns={'Col2':'df'+str(i)})
           for i,xl in enumerate(["set1.xlsx", "set2.xlsx", "set3.xlsx", 
                                  "set4.xlsx", "set5.xlsx", "set6.xlsx"])]

df = reduce(lambda x,y: pd.merge(x, y, on=['Col1'], how='outer'), dfList)

#    Col1      df1      df2
# 0   XYZ  41235.0  43782.0
# 1  OAIS  15123.0      NaN
# 2   ABC  48938.0  33347.0
# 3   KFJ      NaN  21493.0
# 4  SHIZ      NaN  31299.0

Alternatively, use pd.concat and outer join the dataframes horizontally where you need to set Col1 as index:

dfList = [pd.read_excel(xl).rename(columns={'Col2':'df'+str(i)}).set_index('Col1')
           for i,xl in enumerate(["set1.xlsx", "set2.xlsx", "set3.xlsx", 
                                  "set4.xlsx", "set5.xlsx", "set6.xlsx"])]

df2 = pd.concat(dfList, axis=1, join='outer', copy=False)\
                .reset_index().rename(columns={'index':'Col1'})

#    Col1      df1      df2
# 0   ABC  48938.0  33347.0
# 1   KFJ      NaN  21493.0
# 2  OAIS  15123.0      NaN
# 3  SHIZ      NaN  31299.0
# 4   XYZ  41235.0  43782.0