Numpy isnan() 在浮点数组上失败（来自 Pandas 数据帧应用）

Question

I have an array of floats (some normal numbers, some nans) that is coming out of an apply on a pandas dataframe.

For some reason, numpy.isnan is failing on this array, however as shown below, each element is a float, numpy.isnan runs correctly on each element, the type of the variable is definitely a numpy array.

What's going on?!

set([type(x) for x in tester])
Out[59]: {float}

tester
Out[60]: 
array([-0.7000000000000001, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan], dtype=object)

set([type(x) for x in tester])
Out[61]: {float}

np.isnan(tester)
Traceback (most recent call last):

File "<ipython-input-62-e3638605b43c>", line 1, in <module>
np.isnan(tester)

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

set([np.isnan(x) for x in tester])
Out[65]: {False, True}

type(tester)
Out[66]: numpy.ndarray

Answer 1

np.isnan can be applied to NumPy arrays of native dtype (such as np.float64):

In [99]: np.isnan(np.array([np.nan, 0], dtype=np.float64))
Out[99]: array([ True, False], dtype=bool)

but raises TypeError when applied to object arrays:

In [96]: np.isnan(np.array([np.nan, 0], dtype=object))
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

---

Since you have Pandas, you could use pd.isnull instead -- it can accept NumPy arrays of object or native dtypes:

In [97]: pd.isnull(np.array([np.nan, 0], dtype=float))
Out[97]: array([ True, False], dtype=bool)

In [98]: pd.isnull(np.array([np.nan, 0], dtype=object))
Out[98]: array([ True, False], dtype=bool)

Note that None is also considered a null value in object arrays.

Answer 2

A great substitute for np.isnan() and pd.isnull() is

for i in range(0,a.shape[0]):
    if(a[i]!=a[i]):
       //do something here
       //a[i] is nan

since only nan is not equal to itself.

Answer 3

On top of @unutbu answer, you could coerce pandas numpy object array to native (float64) type, something along the line

import pandas as pd
pd.to_numeric(df['tester'], errors='coerce')

Specify errors='coerce' to force strings that can't be parsed to a numeric value to become NaN. Column type would be dtype: float64 , and then isnan check should work

Answer 4

Make sure you import csv file using Pandas

import pandas as pd

condition = pd.isnull(data[i][j])

Answer 5

Just answer this for a reminder of myself. It took me a whole day to solve. After digging deep into the code, I found that in _encodepy.py :

if values.dtype.kind in 'UO':
    # correct branch
else
    # wrong branch, if in this branch whatever data you give it will produce the error
    if np.isnan(known_values).any(): # here is problematic line

so the solution is very simple, just astype your data with np.object