任务期间的 Dask 进度

Question

With dask dataframe using
df = dask.dataframe.from_pandas(df, npartitions=5) series = df.apply(func) future = client.compute(series) progress(future)

In a jupyter notebook I can see progress bar for how many apply() calls completed per partition (eg 2/5).
Is there a way for dask to report progress inside each partition?
Something like tqdm progress_apply() for pandas.

Answer 1

If you mean, how complete each call of func() is, then no, there is no way for Dask to know that. Dask calls python functions which run in their own python thread (python threads cannot be interrupted by another thread), and Dask only knows whether the call is done or not.

You could perhaps conceive of calling a function which has some internal callbacks or other reporting system, but I don't think I've seen anything like that.