Intersection of rows of a Dataframe based on the value in a column in the dataframe

Question

I have a df as shown below. I am trying to find the intersection of rows based on the value of the host column.

host    values 
test    ['A','B','C','D']
test    ['D','E','B','F']
prod    ['1','2','A','D','E']
prod    []
prod    ['2']

the expected output is intersection of the a row with the next row if the host value is same. For the above df, the output would be

test=['B','D'] - intersection of row 1 and 2
prod=[] - intersection of row 3 and 4
prod=[] - intersection of row 4 and 5

the intersection of rows 2 and 3 is not performed as the host column value doesn't match. Any help is appreciated.

The df.to_dict() value is

 {'host': {0: 'test', 1: 'test', 2: 'prod', 3: 'prod', 4: 'prod'},
 'values': {0: ['A', 'B', 'C', 'D'],
  1: ['D', 'E', 'B', 'F'],
  2: ['1', '2', 'A', 'D', 'E'],
  3: [],
  4: ['2']}
 }

Answer 1

Not sure of the structure of expected result, but you could create a column per group of host with shift . then use apply where this new column is notna and do intersection of set s.

df['val_shift'] = df.groupby('host')['values'].shift()
df['intersect'] = df[df['val_shift'].notna()]\
                    .apply(lambda x: list(set(x['values'])&set(x['val_shift'])), axis=1)
print (df)
   host           values        val_shift intersect
0  test     [A, B, C, D]              NaN       NaN
1  test     [D, E, B, F]     [A, B, C, D]    [B, D]
2  host  [1, 2, A, D, E]              NaN       NaN
3  host               []  [1, 2, A, D, E]        []
4  host              [2]               []        []