如果 pandas 中有公共/没有公共列或未知列，如何使用外连接进行合并

Question

Problem Statement: How to perform outer join if we dont have common key (as any additional key appear on

df_a from json_1:

[
    {
        "bookid": "12345",
        "bookname": "who am i"                 
    }
]

df_b from json_2:

[
    {
        "bookid": "12345",
        "bookname": "who am i",        
        "Author" : "asp"        
    }
]

Now i wanted to find difference between these two datafrmes by each key and value (as i need to write output to html table, each column comparison as seperate df )

What I tried below :

df1 = pd.merge(df_a[['bookid']],df_b[['bookid']],left_index=True,right_index=True)
df1['diff'] = np.where((df1['bookid_x']==df1['booid_y']),'No', 'Yes') 


df2 = pd.merge(df_a[['bookname']],df_b[['bookname']],left_index=True,right_index=True)
df2['diff'] = np.where((df2['bookname_x']==df2['bookname_y']),'No', 'Yes')

df3 = What should i write here for that unknown column of Author coming from df2 ? 

with open(r"c:\csv\booktest.html", 'w') as _file:     
     _file.write(df1.to_html(index=False) +  "<br>" + df2.to_html(index=False) + "<br>" + df3.to_html(index=False))
 

The problem is df_b data comes from different source, it might have additional column and values (i dont know what is column name would be in well before hand).

Expected output: (so when i finally compare two df, for example, as Author column is new column i got from df2 which is not present in df_a it should print NaN there

  bookid      bookid       diff
  12345       12345        No

  bookname    bookname     diff
  who am i    who am i     No  
 
  Author      Author       diff
  NaN         asp          Yes  
  
  
  

Answer 1

One way is to align both the data frames so that the columns are same using .align() .

_, df_a = df_b.align(df_a, fill_value=np.NaN)
_, df_b = df_a.align(df_b, fill_value=np.NaN)

Once you do this, both df_a and df_b will have the same columns.

print(df_a)
   Author bookid  bookname
0     NaN  12345  who am i

print(df_b)
  Author bookid  bookname
0    asp  12345  who am i

Now you can apply the logic you have to get df3

df1 = pd.merge(df_a[['bookid']], df_b[['bookid']], left_index=True, right_index=True)
df1['diff'] = np.where((df1['bookid_x']==df1['bookid_y']), 'No', 'Yes')

df2 = pd.merge(df_a[['bookname']], df_b[['bookname']], left_index=True, right_index=True)
df2['diff'] = np.where((df2['bookname_x']==df2['bookname_y']), 'No', 'Yes')

df3 = pd.merge(df_a[['Author']], df_b[['Author']], left_index=True, right_index=True)
df3['diff'] = np.where((df3['Author_x']==df3['Author_y']), 'No', 'Yes')

print(df1)
print(df2)
print(df3)

Result:

  bookid_x bookid_y diff
0    12345    12345   No
  bookname_x bookname_y diff
0   who am i   who am i   No
   Author_x Author_y diff
0       NaN      asp  Yes

EDIT:

Ofcourse, you can put your common statements into a loop - for each column in your df

for col in df_b.columns:
    df_temp = pd.merge(df_a[[col]], df_b[[col]], left_index=True, right_index=True)
    df_temp['diff'] = np.where((df_temp[col+'_x'] == df_temp[col+'_y']), 'No', 'Yes')
    print(df_temp)

Or more effeciently, you can do this - merge both dfs (by all columns) and then find the diff between the pair of columns and export to the html with in the column loop.

df_temp = pd.merge(df_a, df_b, left_index=True, right_index=True)
with open(r"booktest.html", 'w') as _file:
    for col in df_a.columns:
        df_temp[col+'_diff'] = np.where((df_temp[col+'_x'] == df_temp[col+'_y']), 'No', 'Yes')
        _file.write(df_temp[[col + '_x', col + '_y', col + '_diff']].to_html(index=False) + "<br>")
print(df_temp)

You can also do it without .merge , but to get it in the dataframe format in the html, you will have to initialize the dataframe for each column

with open(r"booktest.html", 'w') as _file:
    for col in df_a.columns:
        df_temp = pd.DataFrame()
        df_temp[col + '_x'], df_temp[col + '_y'], df_temp[col + '_diff'] = df_a[col], df_b[col], np.where((df_a[col] == df_b[col]), 'No', 'Yes')
        _file.write(df_temp.to_html(index=False) + "<br>")

Result:

EDIT 2:

Fixed alignment as per comment

text_align = '<style>.dataframe td { text-align: right; }</style>'
with open(r"booktest.html", 'w') as _file:
    for col in df_a.columns:
        df_temp = pd.DataFrame()
        df_temp[col + '_current'], df_temp[col + '_future'], df_temp[col + '_diff'] = df_a[col], df_b[col], np.where((df_a[col] == df_b[col]), 'No', 'Yes')
        _file.write(text_align + df_temp.to_html(index=False) + "<br>")
    print(df_temp)

Result:

EDIT 3

Making a column name as blank if all of its values are NaN s

text_align = '<style>.dataframe td { text-align: right; }</style>'
with open(r"booktest.html", 'w') as _file:
    for col in df_a.columns:
        df_temp = pd.DataFrame()
        df_temp[col + '_current'], df_temp[col + '_future'], df_temp[col + '_diff'] = df_a[col], df_b[col], np.where((df_a[col] == df_b[col]), 'No', 'Yes')
        # check if the column values are all NaN and rename the column name
        [df_temp.rename(columns={c:''}, inplace=True) for c in df_temp.columns if df_temp[c].isnull().all()]
        df_temp.fillna('', inplace=True)
        # set  the display width before writing to html so that blank columns are not squeezed
        with pd.option_context('display.max_colwidth', -1):
            _file.write(text_align+df_temp.to_html(index=False) + "<br>")

Result: