Pandas布尔运算与一次比较与许多比较不一致

Question

I am trying to filter out some rows in my dataframe (with > 400000 rows) where values in one column have the None type. The goal is to leave my dataframe with only rows that have values that are float in the 'Column' column. I plan on doing this by passing in an array of booleans, except that I can't construct my array of booleans properly (they all come back True).

When I run the following operation, given a value of i within the df range, the comparison works:

df.loc[i, 'Column'] != None 

The rows that have a value of None in 'Column' give the results False.

But when I run this operation:

df.loc[0:len(df), 'Column'] != None 

The boolean array comes back as all True.

Why is this? Is this a pandas bug? An edge case? Intended behaviour for reasons I don't understand?

I can think of other ways to construct my boolean array, though this seems the most efficient. But it bothers me that this is the result I am getting.

Answer 1

Here's a reproducible example of what you're seeing:

x = pd.Series([1, None, 3, None, None])

print(x != None)

0    True
1    True
2    True
3    True
4    True
dtype: bool

What's not obvious is behind the scenes Pandas converts your series to numeric and converts those None values to np.nan :

print(x)

0    1.0
1    NaN
2    3.0
3    NaN
4    NaN
dtype: float64

The NumPy array underlying the series can then be held in a contiguous memory block and support vectorised operations. Since np.nan != np.nan by design , your Boolean series will contain only True values, even if you were to test against np.nan instead of None .

For efficiency and correctness, you should use pd.to_numeric with isnull / notnull for checking null values:

print(pd.to_numeric(x, errors='coerce').notnull())

0     True
1    False
2     True
3    False
4    False
dtype: bool