基于百分位数筛选大型数据框的最有效方法

Question

I have a large dataframe, about 5 million rows and 200 columns. I am running the code below to filter out based on percentiles and data types

Code below

col_percentile_filter = 0.98
modeldata_revised_2 = modeldata.loc[:, (modeldata.dtypes!='object') & (modeldata.quantile(col_percentile_filter) >= 1) & (modeldata.min() != modeldata.max())]

The code currently takes a lot of time to run. What is a more efficient way to run this?

Answer 1

When you're running

modeldata_revised_2 = modeldata.loc[:, (modeldata.dtypes!='object') & (modeldata.quantile(col_percentile_filter) >= 1) & (modeldata.min() != modeldata.max())]

You're pretty much computing 3 different dataframes then finding the intersection between them. df.query() is much better for this kind of thing, something like this:

model_min = modeldata.min()
model_max = modeldata.max()
modeldata_revised_2 = modeldata.query("dtypes!='object' & quantile(col_percentile_filter) >= 1 & @model_min != @model_max")

A more comprehensive explanation I found is at: https://jakevdp.github.io/PythonDataScienceHandbook/03.12-performance-eval-and-query.html