'Invalid type comparison' in the code

Question

I have a pandas dataframe which has many columns. These columns may have 3 values - True, False and NaN. I'm replcaing the NaN with the string missing . The sample values for one of my columns is as follows:

ConceptTemp.ix[:,1].values

resulting in:

array([ True, False, False, False,  True,  True,  True,  True, False,  True], dtype=bool)

Note that this particular column had no NaN , and therefore no missing string.

Now I execute the following code:

ConceptTemp.ix[:,1][ConceptTemp.ix[:,1] != 'missing'].values

To get the following exception:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-47-0a0b76cf3ab5> in <module>()
----> 1 ConceptTemp.ix[:,1][ConceptTemp.ix[:,1] != 'missing'].values

E:\Anaconda2\lib\site-packages\pandas\core\ops.pyc in wrapper(self, other, axis)
    724                 other = np.asarray(other)
    725 
--> 726             res = na_op(values, other)
    727             if isscalar(res):
    728                 raise TypeError('Could not compare %s type with Series'

E:\Anaconda2\lib\site-packages\pandas\core\ops.pyc in na_op(x, y)
    680                 result = getattr(x, name)(y)
    681                 if result is NotImplemented:
--> 682                     raise TypeError("invalid type comparison")
    683             except AttributeError:
    684                 result = op(x, y)

TypeError: invalid type comparison

Would someone know how to fix it?

Any pointers would be highly appreciated.

Answer 1

As people have commented, it is a bit weird to combine types in your arrays (ie strings with booleans). You're going to get results where the boolean array may not be what you think it is. But if you absolutely have to, there are a couple of ways you could go about doing this. The first is with isin :

In [40]: ConceptTemp.ix[:,0][~ConceptTemp.ix[:,0].isin(['missing'])].values
Out[40]:
         array([ True, False, False, False,  True,  True,  True,  True, False,  True], dtype=bool)

The second is with apply and lambda

In [41]: ConceptTemp.ix[:,0][ConceptTemp.ix[:,0].apply(lambda x: x != 'missing')].values
Out[41]:
         array([ True, False, False, False,  True,  True,  True,  True, False,  True], dtype=bool)