将繁重的计算功能应用于dask数据帧的最有效方法？

Question

I have a ~6000 rows long dataframe. I need to apply a function (let's call it heavy_func ) to all rows of a specific column, which takes up to a minute per call. The input to the function is a string, and the output an np.array.

I am using dask distributed to set up a cluster on which to perform the apply's. Right now I use 80 workers with 4 cores and 8 GB RAM per worker.

The actual apply step looks like this:

ddf = dd.from_pandas(df, npartitions=100)

ddf['new_col'] = ddf.apply(heavy_func, axis=1, meta=('ddf', 'object'))
df = ddf.compute()

I'm wondering how to maximize efficiency here.

Should I maximize workers or cores? In what ratio should the nr. of partitions and nr. of workers/cores be?

I'm also unsure whether using map_partitions instead of apply would have any benefits.

Answer 1

For processes and threads you may want to read this: https://docs.dask.org/en/latest/best-practices.html#processes-and-threads

It depends on how much your computation releases the GIL.

Using apply vs map_partitions shouldn't make much difference here.