在 Dask dataframe 中查找多列的中值

Question

I have a Dask dataframe with three columns, width, height and length. I need to create a fourth column, which is the median of the three.

My code with regular pandas df doesn't work as median is not a function in Dask.

columns_to_sum = ['weight', 'height', 'length']
df['median'] = df[columns_to_sum].median(axis=1)

Any help is appreciated!

Answer 1

Because the median is the middle value in the ordered set of all values, this is slow and difficult to implement for larger-than-memory data structures.

Dask's dask.DataFrame.quantile implements algorithms for producing approximate quantiles using a variety of algorithms:

df['median'] = df[columns_to_sum].quantile(0.5)

However, as @quasiben pointed out, df[columns_to_sum].mean() will be more efficient even than these approximate algorithms. Also, there are outstanding issues with some of the algorithms, suggesting that dask.DataFrame.quantile can do a very poor job approximating the true quantiles in some edge cases. They're working on it .

Answer 2

While it's true that parallel median is hard, in this case the question asker is asking about median across columns. This is easy because for every row we have all of the data already in memory.

If this doesn't already exist then this should be added to Dask Dataframe. If you want to raise an issue at https://github.com/dask/dask/issues/new that would be welcome.

As a short-term workaround, you can always use Pandas functions and map_partitions

def f(df: pandas.DataFrame, columns: list) -> pandas.DataFrame:
    df = df.copy()  # dask prefers that you not mutate inputs
    df["median"] = df[columns].median(axis=1)

ddf = ddf.map_partitions(f, columns=["a", "b", "c"])