scipy.stats如何处理nans？

Question

I am trying to do some statistics in Python. I have data with several missing values, filled with np.nan , and I am not sure should I remove it manually, or scipy can handle it. So I tried both:

 import scipy.stats, numpy as np
a = [0.75, np.nan, 0.58337, 0.75, 0.75, 0.91663, 1.0, np.nan, 0.663, 0.837,     0.837, 1.0, 0.663, 1.0, 1.0, 0.91663, 0.75, 0.41669, 0.58337, 0.663, 0.75, 0.58337] 
b = [0.837, np.nan, 0.663, 0.58337, 0.75, 0.75, 0.58337, np.nan, 0.166, 0.5,     0.663, 1.0, 0.91663, 1.0, 0.663, 0.75, 0.75, 0.41669, 0.331, 0.25, 1.0, 0.91663]

d_1, d_2 = a,b
wilc1 =  scipy.stats.wilcoxon(d_1, d_2, zero_method = 'pratt')

d_1, d_2 = [], []
for d1, d2 in zip(a, b):
    if np.isnan(d1) or np.isnan(d2):
        pass
    else:
        d_1.append(d1)
        d_2.append(d2)

wilc2 =  scipy.stats.wilcoxon(d_1, d_2, zero_method = 'pratt')
print wilc1
print wilc2

I get two runtime warnigs:

C:\Python27\lib\site-packages\scipy\stats\morestats.py:1963: RuntimeWarning: invalid value encountered in greater
  r_plus = sum((d > 0) * r, axis=0

and two Wilcoxon outputs

(54.0, 0.018545881687477818)
(54.0, 0.056806600853965265)

As you see, I have two similiar test statisitcs (W), and two different P-values. Which is one is correct?

My guess, that Wilcoxon processes missing values correctly during test statistic calculation, but during P-value calculation, it uses len() of all data, not just valid cases. Can this count as bug?

Answer 1

You can not mathematically perform a test statistic based on nan. Unless you find proof/documentation of special treatment of nan, you can not rely on that.

My experience is that in general, even numpy does not treat nan specially, for example for median. Instead the results are whatever they happen to be, as a result of the algorithm implementation.

For example, numpy.median() seems to end up treating nan as inf, placing nan above the median. This is likely just a side effect of the results of a<b comparisons always being false for nan. A similar effect is probably behind your two identical test statistic values W.

Also note: There are a few method variants in numpy, such as http://docs.scipy.org/doc/numpy/reference/generated/numpy.nanmean.html