有没有办法忽略用于屏蔽单独数组的数组中的屏蔽值？

Question

My data is several arrays of data taken of the same length. I am masking one array (y) then using that masked array to mask a 2nd array (x). I mask x to get rid of values indicating equipment error (-9999). I then use np.where() to find out where y is low (1 standard dev below the mean) to mask x in order to see the values of x when y is low.

I have tried changing my mask several times but none of the other numpy masked array operations gave me a different result. I tried to write a logical statement to give me the values when the mask = FALSE but I cannot do that within the np.where() statement.

x = np.array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] )
y = np.array( [ 0, 1, -9999, 3, 4, 5, 6, 7, 8, -9999, 10 ] )

x = np.ma.masked_values( x, -9999 )
y = np.ma.masked_values( y, -9999 )

low_y = ( y.mean() - np.std( y ) )

x_masked = x[ np.where( y < low_y ) ]

When we call x_masked, it returns:

>>>x_masked
masked_array(data=[0, 1, 2, 9],
         mask=False,
   fill_value=-9999)

We expect the mean of x_masked to be 0.5 ( (0 + 1)/2 ) but instead the mean is 3 because of the masked -9999 values ( 2 & 9) that were included in x_masked.

Is there a way to exclude the masked values in order to only get the unmasked values?

Answer 1

I think you'd want to masked x where y != -9999 . If you make this change to your code, it works as you expect.

You could also just use np.where to mask.

x = x[np.where(y != -9999)]
y = y[np.where(y != -9999)]

low_y = ( y.mean() - np.std( y ) )

x_masked = x[np.where( y < low_y)]

print (x_masked)
[0 1]

Answer 2

Since version 1.8 numpy added nanstd and nanmean to handle missing data. In your case since the -9999 is there to indicate error state and by definition I think it is a good use case of numpy.nan

In [76]: y = np.where(y==-9999, np.nan, y)

In [77]: low_y = (np.nanmean(y) - np.nanstd(y))

In [78]: low_y
Out[78]: 1.8177166753143883

In [79]: x_masked = x[ np.where( y < low_y ) ]  # [0, 1]