如何在不同的DataFrames中映射系列

Question

I have two dataframes, one holds the bulk of the data of a dataset and the second one holds some additional data that I got at a later point in time.

Given the example below, I want to replace the values stored in df_main.b with the values found in df_additional.b and I should know which values to use by using the mapping found under column order_id , present in both dataframes.

In [385]: df_main = pd.DataFrame({'order_id':['A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7'], 'b':[1,2,3,4,5,6,7], 'c':np.random.randn(7), 'd':np.random.randn(7)})

In [386]: df_additional = pd.DataFrame({'order_id':['A1', 'A2', 'A3', 'A5', 'A6', 'A7', 'A8'], 'b':['a','b','c','d','e','f','g']})

In [387]: df_main
Out[387]: 
   b         c         d order_id
0  1  0.460474 -1.092239       A1
1  2  0.872538  1.819610       A2
2  3 -0.343626 -2.493006       A3
3  4  0.489427  0.074341       A4
4  5 -1.690572  0.162746       A5
5  6 -0.851540  0.543129       A6
6  7 -0.559258 -0.170457       A7

In [388]: df_additional
Out[388]: 
   b order_id
0  a       A1
1  b       A2
2  c       A3
3  d       A5
4  e       A6
5  f       A7
6  g       A8

Notice how the values in df_main.order_id are not the same as df_additional.order_id .

I would like df_main.b to turn to np.nan for these orders that are present in df_main and not in df_additional (eg 'A4' , thus df_main['b'][3] should turn to np.nan )

I would also like for all those orders that are present in df_additional and not present in df_main to be ignored, nothing new to be added in df_main .

The final output should be:

>>> final_version
   b            c         d order_id
0  a     0.460474 -1.092239       A1
1  b     0.872538  1.819610       A2
2  c    -0.343626 -2.493006       A3
3  NaN   0.489427  0.074341       A4
4  d    -1.690572  0.162746       A5
5  e    -0.851540  0.543129       A6
6  f    -0.559258 -0.170457       A7

Thanks for helping

Edit I have already tried with np.where() with the following results:

In [389]: df_main.b = np.where(df_main.order_id == df_additional.order_id, df_additional.b, np.nan)

In [390]: df_main
Out[390]: 
     b         c         d order_id
0    a  0.460474 -1.092239       A1
1    b  0.872538  1.819610       A2
2    c -0.343626 -2.493006       A3
3  NaN  0.489427  0.074341       A4
4  NaN -1.690572  0.162746       A5
5  NaN -0.851540  0.543129       A6
6  NaN -0.559258 -0.170457       A7

Things go fine until a certain point, but it seems that comparison is made elementwise and therefore fails at some point ( 'A4' != 'A5' ) and from that point on all comparisons fail as well. Is it possible to use some form of isin for all order_id values in df_main , get the index and for that index retrieve the b value (?)

Answer 1

You are looking for merge :

pd.merge(df_additional, df_main, how='right', on='order_id')

#Out[13]:
#   b_x order_id  b_y         c         d
#0    a       A1    1 -2.532221  0.702512
#1    b       A2    2  2.550224 -0.649286
#2    c       A3    3  0.737817  0.999865
#3    d       A5    5 -0.484483  1.153589
#4    e       A6    6  0.526035  0.335695
#5    f       A7    7 -0.901915 -1.312429
#6  NaN       A4    4 -0.905911  0.865345

Answer 2

You can use join() if you make an index from order_id column in df_additional

df_additional.set_index('order_id', inplace=True)    
df_main.join(df_additional, on='order_id', how='left')

Or if you can make indexes from order_id column on both sides, then there is a simple series assignment.

df_main.set_index('order_id', inplace=True)
df_additional.set_index('order_id', inplace=True)    
df_main['b_add'] = df_additional['b']

If you need an example for the second case, here it is 10 Minutes to pandas