Pandas - 根据列内的排名删除单元格

Question

I want to delete values based on their relative rank within their column. Specifically, I want to isolate the X highest and X lowest values within several columns. So if X=2 and my dataframe looks like this:

ID    Val1    Val2    Val3    
001   2       8       14      
002   10      15      8
003   3       1       20
004   11      11      7
005   14      4       19

The output should look like this:

ID    Val1    Val2    Val3    
001   2       NaN     NaN      
002   NaN     15      8
003   3       1       20
004   11      11      7
005   14      4       19

I know that I can make a sub-table to isolate the high and low rank using:

df = df.sort('Column Name')
df2 = df.head(X) # OR: df.tail(X)

And I figure I clear these sub-tables of the values from other columns using:

df2['Other Column'] = np.NaN
df2['Other Column B'] = np.NaN

Then merge the sub-tables back together in a way that replaces NaN values when there is data in one of the tables. I tried:

df2.update(df3) # df3 is a sub-table made the same way as df2 using a different column

Which only updated rows already present in df2.

I tried:

out = pd.merge(df2, df3, how='outer')

which gave me separate rows when a row appeared in both df2 and d3

I tried:

out = df2.combine_first(df3)

which over-wrote numerical values with found NaN values in some cases making it unsuitable.

There must be a way to do this: I want to the original dataframe with NaN values plugged in whenever a value is not among the X highest or X lowest values in that column.

Answer 1

Interesting question, you can get the index of the values of each columns in the sorted values of each columns (here in the mask DataFrame ), and then keep the values that have the index within you defined boundary.

In [98]:
print df
    Val1  Val2  Val3
ID                  
1      2     8    14
2     10    15     8
3      3     1    20
4     11    11     7
5     14     4    19
In [99]:

mask = df.apply(lambda x: np.searchsorted(sorted(x),x))
print mask
    Val1  Val2  Val3
ID                  
1      0     2     2
2      2     4     1
3      1     0     4
4      3     3     0
5      4     1     3
In [100]:

print (mask<=1)|(mask>=(len(mask)-2))
     Val1   Val2   Val3
ID                     
1    True  False  False
2   False   True   True
3    True   True   True
4    True   True   True
5    True   True   True
In [101]:

print df.where((mask<=1)|(mask>=(len(mask)-2)))
    Val1  Val2  Val3
ID                  
1      2   NaN   NaN
2    NaN    15     8
3      3     1    20
4     11    11     7
5     14     4    19