大熊猫加入DF-合并与加入不同的语义

Question

I want to join 2 DF in pandas. Some columns are int or float, others are categories. (not enforcing the same cat code/index for categories from A and B df) Their common columns are a list of float and category columns of size 8.

Joining via

df_a.merge(df_b, how='inner'), on=join_columns )

will return no result at all. And joining via

df_a.join(df_b, lsuffix='_l', rsuffix='_r')

Seems to work.

But I am a bit confused why one failed and if I should not cast all columns to object in order to prevent joining by cat codes which could be wrong.

Ie if left is chosen as join method for merge , joined columns will only contain NAN values. Unfortunately, I am not really sure how t build an useful minimal example.

edit

here a sample

import pandas as pd

raw_data = {
        'subject_id': ['1', '2', '3', '4', '5'],
        'name': ['A', 'B', 'C', 'D', 'E'],
        'nationality': ['DE', 'AUT', 'US', 'US', 'US'],
        'age_group' : [1, 2, 1, 3, 1]}
df_a = pd.DataFrame(raw_data, columns = ['subject_id', 'name', 'nationality', 'age_group'])
df_a.nationality = df_a.nationality.astype('category')
df_a


raw_data = {
        'subject_id': ['1', '2', '3' ],
        'name': ['Billy', 'Brian', 'Bran'],
        'nationality': ['DE', 'US', 'US'],
        'age_group' : [1, 1, 3],
        'average_return_per_group' : [1.5, 2.3, 1.4]}
df_b = pd.DataFrame(raw_data, columns = ['subject_id', 'name', 'nationality', 'age_group', 'average_return_per_group'])
df_b.nationality = df_b.nationality.astype('category')
df_b


# some result is joined
df_a.join(df_b, lsuffix='_l', rsuffix='_r') 

# this *fails* as only NULL values joined, or nor result for inner join
df_a.merge(df_b, how='left', on=['nationality', 'age_group'])

Answer 1

join joins per default along the indexes, and merge along the columns with the same names.

Check this:

In [115]: df_a.join(df_b, lsuffix='_l', rsuffix='_r')
Out[115]:
  subject_id_l name_l nationality_l  age_group_l subject_id_r name_r nationality_r  age_group_r average_returns_per_group
0            1      A            DE            1            1  Billy            DE          1.0                       NaN
1            2      B           AUT            2            2  Brian            US          1.0                       NaN
2            3      C            US            1            3   Bran            US          3.0                       NaN
3            4      D            US            3          NaN    NaN           NaN          NaN                       NaN
4            5      E            US            1          NaN    NaN           NaN          NaN                       NaN

let's set ['a','b','c'] as an index in df_b and try to join it again - you'll see only NaN 's in all *_r columns:

In [116]: df_a.join(df_b.set_index(pd.Index(['a','b','c'])), lsuffix='_l', rsuffix='_r')
Out[116]:
  subject_id_l name_l nationality_l  age_group_l subject_id_r name_r nationality_r  age_group_r average_returns_per_group
0            1      A            DE            1          NaN    NaN           NaN          NaN                       NaN
1            2      B           AUT            2          NaN    NaN           NaN          NaN                       NaN
2            3      C            US            1          NaN    NaN           NaN          NaN                       NaN
3            4      D            US            3          NaN    NaN           NaN          NaN                       NaN
4            5      E            US            1          NaN    NaN           NaN          NaN                       NaN

In [117]: df_b.set_index(pd.Index(['a','b','c']))
Out[117]:
  subject_id   name nationality  age_group average_returns_per_group
a          1  Billy          DE          1                       NaN
b          2  Brian          US          1                       NaN
c          3   Bran          US          3                       NaN

UPDATE: IMO merge works as expected (described in docs)

In [151]: df_a.merge(df_b, on=['nationality', 'age_group'], how='left', suffixes=['_l','_r'])
Out[151]:
  subject_id_l name_l nationality  age_group subject_id_r name_r  average_return_per_group
0            1      A          DE          1            1  Billy                       1.5
1            2      B         AUT          2          NaN    NaN                       NaN
2            3      C          US          1            2  Brian                       2.3
3            4      D          US          3            3   Bran                       1.4
4            5      E          US          1            2  Brian                       2.3

Answer 2

我认为主要区别是join具有默认的left join和merge inner join.