从具有字符串值和数字值的numpy数组中删除NaN

Question

I have a (M x N) numpy array, which contains string values, numerical values and nans. I want to drop the rows which contain NaN values. I've tried:

arr[~np.isnan(arr)]

however i get the error:

TypeError: ufunc 'isnan' not supported for the input types, and the inputs 
could not be safely coerced to any supported types according to the casting rule ''save''

Solution that I used:

# get column with NaNs, find row index, store in list
nan_idx = []
for v,m in enumerate(arr[:,row]):
    if np.isnan(m):
        nan_idx.append(v)

# separate columns with strings and non strings
numeric_cols = arr[:,:some_idx]
non_numeric_cols = arr[:,other_idx:]

# remove the nans
numeric_cols = numeric_cols[~np.isnan(numeric_cols).any(axis=1)]
non_numeric_cols = np.delete(non_numeric_cols, nan_idx, 0)

Answer 1

I get your error if I make an object dtype array:

In [112]: arr=np.ones((3,2),object)
In [113]: arr
Out[113]: 
array([[1, 1],
       [1, 1],
       [1, 1]], dtype=object)
In [114]: np.isnan(arr)
...
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

That dtype is the only one that can mix numbers, strings and np.nan (which is a float). You can't do a lot of whole-array operations with this.

I can't readily test your solution because several variables are unknown.

With a more general arr , I don't see how you can remove a row without iterating over both rows and cols, testing whether each value is numeric, and if numeric isnan . np.isnan is picky and can only operate on a float.

As mentioned in the 'possible duplicate' pandas isnull is more general.

So basically two points:

• what's a good general test that can handle strings as well as numbers

• can you get around a full iteration, assuming the array is dtype object.

np.isnan on arrays of dtype "object" My solution here is to do a list comprehension to loop over a 1d array.

From that I can test each element of arr with:

In [125]: arr
Out[125]: 
array([['str', 1],
       [nan, 'str'],
       [1, 1]], dtype=object)
In [136]: for row in arr:
     ...:     for col in row:
     ...:         print(np.can_cast(col,float) and np.isnan(col))
False
False
True
False
False
False

Answer 2

One solution is you can use np.sum() to sum each row up. because nan + any float = nan, so that you can get which lines incluede nan value.

np.sum(arr,axis = 1)
rowsWithoutNaN = [ not(np.isnan(i)) for i in b]
result = np.array( [val for shouldKeep, val in zip(rowsWithoutNaN,arr) if shouldKeep])