在pandas系列上使用apply方法获取TypeError'Series'对象是可变的，因此不能将它们散列

Question

I have two data frames D1 and D2 . What I want to achieve is for any column pairs in D1 and D2 which are non-int and non-float type, I want to compute a distance metric using the formula

 |A intersect B|/ |A union B|

I first defined a function

def jaccard_d(series1, series2):
    if (series1.dtype is not (pd.np.dtype(int) or pd.np.dtype(float))) and     (series2.dtype is not (pd.np.dtype(int) or pd.np.dtype(float))):
        series1 = series1.drop_duplicates()
        series2 = series2.drop_duplicates()
        return len(set(series1).intersection(set(series2)))     /len(set(series1).union(set(series2)))
    else:
        return np.nan

Then what I did is to first loop over all columns in D1 , then for each fixed column in D1 , I use apply on my jaccard_d function. I try to avoid writing 2 layer loops. May be there is a better way without wrting any loops?

DC = dict.fromkeys(list(D1.columns))
INN = list(D2.columns)
for col in D1:
    DC[col] = dict(zip(INN, D2.apply(jaccard_d,D1[col])))

First, I am not sure whether I use the apply function correctly, ie, my jaccard_d function takes 2 series as input, but here for each iteration, I have D1[col] as one series, and I want to use apply to apply D1[col] to all columns of D2

Second, I get this error "'Series' objects are mutable, thus they cannot be hashed", which I don't quite understand. Any comments are appreciated.

I tried to just write a 2-layer loop and use my function jaccard_d to do that. It works. But I want to write more efficient code.

Answer 1

So after floundering around, and finding exactly where the error occurs, and checking the apply docs, I've deduced that you need to call apply thusly:

 D2.apply(jaccard_d, args=(D1[col],))

Instead you were using

 D2.apply(jaccard_d, axis=D1[col])

==================

I can reproduce your error message with a simple dataframe:

In [589]: df=pd.DataFrame(np.arange(12).reshape(6,2))
In [590]: df
Out[590]: 
    0   1
0   0   1
1   2   3
2   4   5
3   6   7
4   8   9
5  10  11

Putting a Series in set works, just as if we'd put a list in set :

In [591]: set(df[0]).union(set(df[1]))
Out[591]: {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}

But if I try to put a list containing a Series in the set I get your error.

In [592]: set([df[0]])
....
TypeError: 'Series' objects are mutable, thus they cannot be hashed

If the problem isn't with the the set expressions then it occurs in the dict() one.

You did not specify where the error occurs, nor have you given a MVCe.

(but as it turns out this is a deadend)

========================

OK, simulating your code:

In [606]: DC=dict.fromkeys(list(df.columns))
In [607]: DC
Out[607]: {0: None, 1: None}
In [608]: INN=list(df.columns)
In [609]: INN
Out[609]: [0, 1]
In [610]: for col in df:
     ...:     dict(zip(INN, df.apply(jaccard_d, df[col])))
    ....
----> 2     dict(zip(INN, df.apply(jaccard_d, df[col])))


/usr/local/lib/python3.5/dist-packages/pandas/core/frame.py in apply(self, func, axis, broadcast, raw, reduce, args, **kwds)
   ...
-> 4125         axis = self._get_axis_number(axis)

/usr/local/lib/python3.5/dist-packages/pandas/core/generic.py in _get_axis_number(self, axis)
    326 
    327     def _get_axis_number(self, axis):
--> 328         axis = self._AXIS_ALIASES.get(axis, axis)
    ....        

TypeError: 'Series' objects are mutable, thus they cannot be hashed

So the problem is in

df.apply(jaccard_d, df[0])

The problem has nothing to do with jaccard_d . It occurs if I replace it with simple

def foo(series1, series2):
    print(series1)
    print(series2)
    return 1

======================

But look at the docs for apply

df.apply(func, axis=0, broadcast=False, raw=False, reduce=None, args=(), **kwds)

The 2nd argument, if not keyword, is the axis number. So we have been trying to use a Series as the axis number! No wonder it objects! That should have been obvious if I'd read the error trace more carefully.

Leaving the default axis=0 , lets pass the other Series as args :

In [632]: df.apply(jaccard_d,args=(df[1],))
Out[632]: 
0    0.0
1    1.0
dtype: float64

or in your loop:

In [643]: for col in df:
     ...:     DC[col] = dict(zip(INN, df.apply(jaccard_d,args=(df[col],))))  
In [644]: DC
Out[644]: {0: {0: 1.0, 1: 0.0}, 1: {0: 0.0, 1: 1.0}}