[英]Python comparison ignoring nan
While nan == nan
is always False
, in many cases people want to treat them as equal, and this is enshrined in pandas.DataFrame.equals
:虽然
nan == nan
总是False
,但在许多情况下人们希望将它们视为平等,这在pandas.DataFrame.equals
中pandas.DataFrame.equals
:
NaNs in the same location are considered equal.
同一位置的 NaN 被认为是相等的。
Of course, I can write当然,我可以写
def equalp(x, y):
return (x == y) or (math.isnan(x) and math.isnan(y))
However, this will fail on containers like [float("nan")]
and isnan
barfs on non-numbers (so the complexity increases ).但是,这将在非数字上的
[float("nan")]
和isnan
barfs 等容器上失败(因此复杂性增加)。
So, what do people do to compare complex Python objects which may contain nan
?那么,人们如何比较可能包含
nan
复杂 Python 对象呢?
PS .附注。 Motivation: when comparing two rows in a pandas
DataFrame
, I would convert them into dict
s and compare dicts element-wise.动机:当比较 pandas
DataFrame
两行时,我会将它们转换为dict
并按元素比较 dicts 。
PPS .聚苯乙烯。 When I say " compare ", I am thinking
diff
, not equalp
.当我说“比较”时,我在想
diff
,而不是equalp
。
Suppose you have a data-frame with nan
values:假设您有一个带有
nan
值的数据框:
In [10]: df = pd.DataFrame(np.random.randint(0, 20, (10, 10)).astype(float), columns=["c%d"%d for d in range(10)])
In [10]: df.where(np.random.randint(0,2, df.shape).astype(bool), np.nan, inplace=True)
In [10]: df
Out[10]:
c0 c1 c2 c3 c4 c5 c6 c7 c8 c9
0 NaN 6.0 14.0 NaN 5.0 NaN 2.0 12.0 3.0 7.0
1 NaN 6.0 5.0 17.0 NaN NaN 13.0 NaN NaN NaN
2 NaN 17.0 NaN 8.0 6.0 NaN NaN 13.0 NaN NaN
3 3.0 NaN NaN 15.0 NaN 8.0 3.0 NaN 3.0 NaN
4 7.0 8.0 7.0 NaN 9.0 19.0 NaN 0.0 NaN 11.0
5 NaN NaN 14.0 2.0 NaN NaN 0.0 NaN NaN 8.0
6 3.0 13.0 NaN NaN NaN NaN NaN 12.0 3.0 NaN
7 13.0 14.0 NaN 5.0 13.0 NaN 18.0 6.0 NaN 5.0
8 3.0 9.0 14.0 19.0 11.0 NaN NaN NaN NaN 5.0
9 3.0 17.0 NaN NaN 0.0 NaN 11.0 NaN NaN 0.0
And you want to compare rows, say, row 0 and 8. Then just use fillna
and do vectorized comparison:并且您想比较行,例如第 0 行和第 8 行。然后只需使用
fillna
并进行矢量化比较:
In [12]: df.iloc[0,:].fillna(0) != df.iloc[8,:].fillna(0)
Out[12]:
c0 True
c1 True
c2 False
c3 True
c4 True
c5 False
c6 True
c7 True
c8 True
c9 True
dtype: bool
You can use the resulting boolean array to index into the columns, if you just want to know which columns are different:如果您只想知道哪些列不同,您可以使用生成的布尔数组对列进行索引:
In [14]: df.columns[df.iloc[0,:].fillna(0) != df.iloc[8,:].fillna(0)]
Out[14]: Index(['c0', 'c1', 'c3', 'c4', 'c6', 'c7', 'c8', 'c9'], dtype='object')
I assume you have array-data or can at least convert to a numpy array?我假设您有数组数据或至少可以转换为 numpy 数组?
One way is to mask all the nans using a numpy.ma
array, then comparing the arrays.一种方法是使用
numpy.ma
数组屏蔽所有numpy.ma
,然后比较数组。 So your starting situation would be sth.所以你的开始情况将是…… like this
像这样
import numpy as np
import numpy.ma as ma
arr1 = ma.array([3,4,6,np.nan,2])
arr2 = ma.array([3,4,6,np.nan,2])
print arr1 == arr2
print ma.all(arr1==arr2)
>>> [ True True True False True]
>>> False # <-- you want this to show True
Solution:解决方法:
arr1[np.isnan(arr1)] = ma.masked
arr2[np.isnan(arr2)] = ma.masked
print arr1 == arr2
print ma.all(arr1==arr2)
>>> [True True True -- True]
>>> True
Here's a function that recurses into a data structure replacing nan
values with a unique string.这是一个递归到数据结构中的函数,用唯一的字符串替换
nan
值。 I wrote this for a unit test that compares data structures that may contain nan
.我写这个是为了一个单元测试,它比较可能包含
nan
数据结构。
It's only designed for data structures made of dict
and list
, but it's easy to see how to expand it.它只是
dict
和list
组成的数据结构设计的,但很容易看到如何扩展它。
from math import isnan
from uuid import uuid4
from typing import Union
NAN_REPLACEMENT = f"THIS_WAS_A_NAN{uuid4()}"
def replace_nans(data_structure: Union[dict, list]) -> Union[dict, list]:
if isinstance(data_structure, dict):
iterme = data_structure.items()
elif isinstance(data_structure, list):
iterme = enumerate(data_structure)
else:
raise ValueError(
"replace_nans should only be called on structures made of dicts and lists"
)
for key, value in iterme:
if isinstance(value, float) and isnan(value):
data_structure[key] = NAN_REPLACEMENT
elif isinstance(value, dict) or isinstance(value, list):
data_structure[key] = replace_nans(data_structure[key])
return data_structure
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.