Pandas 通过在不使用 iterrows 的情况下查询其他数据帧来创建新的 dataframe

Question

I have two huge dataframes that both have the same id field. I want to make a simple summary dataframe where I show the maximum of specific columns. I understand iterrows() is frowned upon, so are a couple one-liners to do this? I don't understand lambda/apply very well, but maybe this would work here.

Stand-alone example

import pandas as pd

myid = [1,1,2,3,4,4,5]
name =['A','A','B','C','D','D','E']
x = [15,12,3,3,1,4,8]
df1 = pd.DataFrame(list(zip(myid, name, x)), 
                  columns=['myid', 'name', 'x'])
display(df1)

myid = [1,2,2,2,3,4,5,5]
name =['A','B','B','B','C','D','E','E']
y = [9,6,3,4,6,2,8,2]
df2 = pd.DataFrame(list(zip(myid, name, y)), 
                  columns=['myid', 'name', 'y'])
display(df2)

mylist = df['myid'].unique()
df_summary = pd.DataFrame(mylist, columns=['MY_ID'])
## do work here...

Desired output

Answer 1

you can try concat+groupby.max

out = (pd.concat((df1,df2),sort=False).groupby(['myid','name']).max()
         .add_prefix("Max_").reset_index())

---

   myid name  Max_x  Max_y
0     1    A   15.0    9.0
1     2    B    3.0    6.0
2     3    C    3.0    6.0
3     4    D    4.0    2.0
4     5    E    8.0    8.0

Answer 2

• merge()
• named aggregations

df1.merge(df2, on=["myid","name"], how="outer")\
.groupby(["myid","name"], as_index=False).agg(MAX_X=("x","max"),MAX_Y=("y","max"))

	myid	name	MAX_X	MAX_Y
0	1	A	15	9
1	2	B	3	6
2	3	C	3	6
3	4	D	4	2
4	5	E	8	8

updated

• you have noted that your data frames are large and solution is giving you OOM
• logically aggregate first, then merge will use less memory

pd.merge(
    df1.groupby(["myid","name"],as_index=False).agg(MAX_X=("x","max")),
    df2.groupby(["myid","name"],as_index=False).agg(MAX_Y=("y","max")),
    on=["myid","name"]
)