包含列表的熊猫数据框列，获取两个连续行的交集

Question

I have following dataframe:

timestmp              coulmnis                                                
2015-10-15 18:24:00  set([a,b,c,d,e,f])
2015-10-15 18:27:00  set([a,b,g,h,i])
2015-10-15 18:30:00  set([g,h,j,k,l])
2015-10-15 18:33:00  set([a,b,g,h,j,k,l])
2015-10-15 18:36:00  set([d,e,j,k])

I want to check how many elemnets in a row is same as previous row.My output should look like this:

timestmp              coulmnis                   count_sameAsPrevious                          
2015-10-15 18:24:00  set([a,b,c,d,e,f])          0
2015-10-15 18:27:00  set([a,b,g,h,i])            2
2015-10-15 18:30:00  set([g,h,j,k,l])            2
2015-10-15 18:33:00  set([a,b,g,h,j,k,l])        5
2015-10-15 18:36:00  set([d,e,j,k])              2

What is the most efficient way so that I can avoid a for loop.Any help appreciated!!

EDIT:

df['shiftedColumn'] = df.columnis.shift(1)
df = df.dropna()

Now I want to use len(filter(y.__contains__,x)) to get no of same elements in two columns which contains set .

Answer 1

You can do this using DataFrame.shift() to shift the rows by one column and then rename the coulmnis column to something else, then reset index and merge the dataframes on timestmp and then use apply() on the DataFrame. Example (in one line) -

df['count'] = df.reset_index().merge(df.shift(1).reset_index().rename(columns={'coulmnis':'newcol'})) \
                .set_index('timestmp').apply((lambda x: len(x['coulmnis'] & x['newcol']) if pd.notnull(x['newcol']) else 0),axis=1)

Example in a more readable way -

mergedf = df.shift(1).reset_index().rename(columns={'coulmnis':'newcol'})
newdf = df.merge(mergedf).set_index('timestmp')
df['count'] = newdf.apply((lambda x: len(x['coulmnis'] & x['newcol']) if pd.notnull(x['newcol']) else 0),axis=1)

---

Demo -

In [36]: df
Out[36]:
                                       coulmnis
timestmp
2015-10-15 18:24:00     set([f, b, c, e, d, a])
2015-10-15 18:27:00        set([g, b, i, a, h])
2015-10-15 18:30:00        set([l, g, k, j, h])
2015-10-15 18:33:00  set([b, j, h, k, a, l, g])
2015-10-15 18:36:00           set([d, e, k, j])

In [38]: df['count'] = df.reset_index().merge(df.shift(1).reset_index().rename(columns={'coulmnis':'newcol'})) \
   ....:                 .set_index('timestmp').apply((lambda x: len(x['coulmnis'] & x['newcol']) if pd.notnull(x['newcol']) else 0),axis=1)

In [39]: df
Out[39]:
                                       coulmnis  count
timestmp
2015-10-15 18:24:00     set([f, b, c, e, d, a])      0
2015-10-15 18:27:00        set([g, b, i, a, h])      2
2015-10-15 18:30:00        set([l, g, k, j, h])      2
2015-10-15 18:33:00  set([b, j, h, k, a, l, g])      5
2015-10-15 18:36:00           set([d, e, k, j])      2

Answer 2

My solution:

df = pandas.DataFrame({'sets': [set(['a','b','c','d','e','f']), set(['a','b','g','h','i']), set(['g','h','j','k','l']), set(['a','b','g','h','j','k','l'])]})
df['sets_temp'] = pandas.Series([])
df['sets_temp'][1:] = df['sets'][:-1]
df['count'] = pandas.Series([])
df['count'][1:] = df[1:].apply(lambda row: len(row['sets'] & row['sets_temp']), axis=1)
df['count'][:1] = 0
df = df.drop('sets_temp', axis=1)

Output:

>>> df
                         sets  count
0     set([b, c, d, e, a, f])      0
1        set([b, h, i, a, g])      2
2        set([j, h, l, k, g])      2
3  set([j, b, h, k, l, a, g])      5

Actually apply() function is a wrapper on for loop , so the efficience of apply() is the same, but it looks like there is no chance to escape of using for loop-like method.