根据 pandas DataFrame 中的一对列从辅助表中查找交集的最有效方法是什么？

Question

I have 3 DataFrames in Pandas:

UserItem is a DataFrame of users and items that the users chose, with 2 columns, User and Item.

UserTag is a DataFrame of users and tags, with 2 columns, User and Tag.

ItemTag is a DataFrame of items and tags, with 2 columns, Item and Tag.

UserItem_df = pd.DataFrame({'user': ['A', 'B', 'B']      ,  'item': ['i', 'j', 'k']})
UserTag_df  = pd.DataFrame({'user': ['A', 'B']           ,  'tag' : ['T', 'R']})
ItemTag_df  = pd.DataFrame({'item': ['i', 'j', 'k', 'k'] ,  'tag' : ['T', 'S', 'T', 'R']})

I want to compute, for each (user, item) pair in UserItem, the size of the intersection (and union as well.) of the tags of that user with the tags of that item.

Answer_df = pd.DataFrame({'user': ['A', 'B', 'B']  , 'item': ['i', 'j', 'k'], 'intersection':  [1, 0, 1], 'union' : [1, 2, 2]})

What's the most efficient way to do this? These are DataFrames with 30M rows ( UserItem_df ), and about 500k rows for the other two. The product set of all possible (user, item) pairs is about 30 billion - I don't need the intersection and unions for all possible pairs, just the ones in the UserItem dataframe.

Answer 1

Use:

# step 1:
df1 = pd.merge(UserItem_df, UserTag_df, on='user')

# step 2:
df2 = pd.merge(UserItem_df, ItemTag_df, on='item')

# step 3
df3 = pd.concat([df1, df2], ignore_index=True)

# step 4
df3 = (
    df3.groupby(['user', 'item'])['tag']
    .agg(intersection='count', union='nunique')
    .reset_index()
)
df3['intersection'] -= df3['union']

Steps:

# step 1: df1
  user item tag
0    A    i   T
1    B    j   R
2    B    k   R

# step 2: df2
  user item tag
0    A    i   T
1    B    j   S
2    B    k   T
3    B    k   R

# step 3: df3
  user item tag
0    A    i   T
1    B    j   R
2    B    k   R
3    A    i   T
4    B    j   S
5    B    k   T
6    B    k   R

# step 4: df3
  user item  intersection  union
0    A    i             1      1
1    B    j             0      2
2    B    k             1      2