连接两个数据帧，其中列值（一组）是另一个数据帧的子集

Question

I have two data frames:

df1 = pd.DataFrame([[set(['foo', 'baz'])],
                    [set(['bar', 'baz'])]], columns=['items'])



    items
0   {foo, baz}
1   {bar, baz}

df2 = pd.DataFrame([[set(['bar', 'baz', 'foo']), 1],
                    [set(['bar', 'baz', 'foo']), 2],
                    [set(['bar', 'baz', 'foo']), 3],
                    [set(['one', 'two', 'bar']), 2]], columns=['items', 'other'])



    items           other
0   {foo, bar, baz} 1
1   {foo, bar, baz} 2
2   {foo, bar, baz} 3
3   {two, one, bar} 2

The goal is to join df2 with df1 where the values in df1.items are a subset of df2.items . Both columns are a set()

For context, this is to join association rules with customer purchases after implementing the apriori algorithm.

Adding expected output:

df3 = pd.DataFrame([[[set(['foo', 'baz'])], set(['bar', 'baz', 'foo']), 1],
                    [[set(['foo', 'baz'])], set(['bar', 'baz', 'foo']), 2],
                    [[set(['foo', 'baz'])], set(['bar', 'baz', 'foo']), 3],
                    [[set(['bar', 'baz'])], None, None]], columns=['items', 'items', 'other'])


    items           items           other
0   [{foo, baz}]    {foo, bar, baz} 1.0
1   [{foo, baz}]    {foo, bar, baz} 2.0
2   [{foo, baz}]    {foo, bar, baz} 3.0
3   [{bar, baz}]    None    NaN

Answer 1

Create your dataframes

import pandas as pd

df1 = pd.DataFrame({'key': [1, 1],
                    'id': [0, 1],
                    'items': [set(['foo', 'baz']), set(['bar', 'baz'])]})

df2 = pd.DataFrame({'key': [1, 1, 1, 1],
                    'items': [set(['bar', 'baz', 'foo']), set(['bar', 'baz', 'foo']), set(['bar', 'baz', 'foo']), set(['one', 'two', 'bar'])],
                    'other': [1, 2, 3, 2]
                   })

then make a cartesian product

merged_df = df1.merge(df2, on='key')
merged_df

   key  id     items_x          items_y  other
0    1   0  {baz, foo}  {foo, baz, bar}      1
1    1   0  {baz, foo}  {foo, baz, bar}      2
2    1   0  {baz, foo}  {foo, baz, bar}      3
3    1   0  {baz, foo}  {one, bar, two}      2
4    1   1  {baz, bar}  {foo, baz, bar}      1
5    1   1  {baz, bar}  {foo, baz, bar}      2
6    1   1  {baz, bar}  {foo, baz, bar}      3
7    1   1  {baz, bar}  {one, bar, two}      2

define your custom function and see if it works in one case

def check_if_all_in_list(list1, list2):
    return all(elem in list2 for elem in list1)

check_if_all_in_list(merged_df['items_x'][0], merged_df['items_y'][0])
True

Create your match

merged_df['check'] = merged_df.apply(lambda row: check_if_all_in_list(row['items_x'], row['items_y']), axis=1)
merged_df

   key  id     items_x          items_y  other  check
0    1   0  {baz, foo}  {foo, baz, bar}      1   True
1    1   0  {baz, foo}  {foo, baz, bar}      2   True
2    1   0  {baz, foo}  {foo, baz, bar}      3   True
3    1   0  {baz, foo}  {one, bar, two}      2  False
4    1   1  {baz, bar}  {foo, baz, bar}      1   True
5    1   1  {baz, bar}  {foo, baz, bar}      2   True
6    1   1  {baz, bar}  {foo, baz, bar}      3   True
7    1   1  {baz, bar}  {one, bar, two}      2  False

now filter out what you don't want

mask = (merged_df['check']==True)
merged_df[mask]

   key  id     items_x          items_y  other  check
0    1   0  {baz, foo}  {foo, baz, bar}      1   True
1    1   0  {baz, foo}  {foo, baz, bar}      2   True
2    1   0  {baz, foo}  {foo, baz, bar}      3   True
4    1   1  {baz, bar}  {foo, baz, bar}      1   True
5    1   1  {baz, bar}  {foo, baz, bar}      2   True
6    1   1  {baz, bar}  {foo, baz, bar}      3   True

Answer 2

In case if you want to simply filter df2 as per the condition (so kind of like select ... from table where X in (select ...) ) - you can do:

df2.loc[df2["items"].apply(lambda x: any(el.intersection(x)==el for el in df1["items"].tolist()))]

Output:

   items                other
0  {foo, baz, bar}      1
1  {foo, baz, bar}      2
2  {foo, baz, bar}      3

To achieve "left join"-like effect:

import numpy as np

df2["match"]=df2["items"].apply(lambda x: any(el.intersection(x)==el for el in df1["items"].tolist()))

df2.loc[~df2["match"], ["other"]]=np.nan

df2.drop(columns="match", inplace=True)

Output:

   items              other
0  {bar, baz, foo}    1.0
1  {bar, baz, foo}    2.0
2  {bar, baz, foo}    3.0
3  {two, bar, one}    NaN