Selecting common elements in dataframes Python

Question

I have 3 DFs, and I want to find all cells with common directions (ie always positive or always negative, across DFs):

test=pd.DataFrame([[0,1,0,3],
                   [-1,3,0,2],
                   [2,1.5,-3,1]],
                  columns=['a','b','c','d']
                   )
test2=pd.DataFrame([[1,1,0,2],
                   [1,-3,0,1],
                   [2,1.5,-2,1]],
                  columns=['a','b','c','d']
                   )
test3=pd.DataFrame([[1,2,0,2],
                   [1,-2,0,1],
                   [2,1.5,-2,1]],
                  columns=['a','b','c','d']
                   )

The outcome should be the 3 dataframes, where elements that are not consistent show NAs. For instance, for test1 it would be:

test=pd.DataFrame([[NA,1,NA,3],
                   [NA,NA,NA,2],
                   [2,1.5,-3,1]],
                  columns=['a','b','c','d']
                   )

Note that 0 is not considered (ie, leads to NA). I can do this cell by cell, but I'm wondering if this is possible to do in the entire dataframes at once?

I was trying to do ((test>0)&(test1>0)&(test2>0)) and this works, but I cannot merge this with the negatives.

Thanks so much in advance

Answer 1

A slightly different approach - you could stack the underlying arrays together, use np.sign , then sum and reduce over the added dimension to generate a mask for df.where .

In [58]: m, n = test.shape

In [59]: signs = np.sign(np.dstack((test, test2, test3)))

In [60]: mask = np.abs(np.sum(signs, -1)) == m

In [61]: test.where(mask)
Out[61]: 
     a    b    c  d
0  NaN  1.0  NaN  3
1  NaN  NaN  NaN  2
2  2.0  1.5 -3.0  1

Answer 2

You can use np.sign and addition with a equality test then where to do this:

 test.where(np.sign(test).add(np.sign(test2)).add(np.sign(test3)).abs() == 3)

Output:

     a    b    c  d
0  NaN  1.0  NaN  3
1  NaN  NaN  NaN  2
2  2.0  1.5 -3.0  1