将 tqdm 与延迟执行与 python 中的 dask 相结合

Question

tqdm and dask are both amazing packages for iterations in python. While tqdm implements the needed progress bar, dask implements the multi-thread platform and they both can make iteration process less frustrating. Yet - I'm having troubles to combine them both together.

For example, the following code implements a delayed execution in dask , with tqdm.trange progress bar. The thing is that since the delayed is performed quickly, the progress bar ends immediately, while the real computation-time effort is done during the compute part.

from dask import delayed,compute
from tqdm import trange
from time import sleep

ct = time()
result= []

def fun(x):
    sleep(x)
    return x

for i in trange(10):
    result.append(delayed(fun)(i))

print compute(result)

How can I attach the progress bar to the actual execution in compute command?

Answer 1

Consider Dask's progress bar

from dask.diagnostics import ProgressBar

with ProgressBar():
    compute(result)

Build a diagnostic of your own

You can use this plugin architecture to get a signal at the end of every task. http://dask.pydata.org/en/latest/diagnostics.html

Here is an example of someone doing exactly this: https://github.com/tqdm/tqdm/issues/278

Answer 2

Based on :

Dask Integration

from tqdm.dask import TqdmCallback

with TqdmCallback(desc="compute"):
    ...
    arr.compute()

# or use callback globally
cb = TqdmCallback(desc="global")
cb.register()
arr.compute()

Applied to the code in the question:

from dask import delayed,compute
from tqdm.auto import tqdm
# from tqdm import trange
from time import sleep

from tqdm.dask import TqdmCallback

# ct = time()
result= []

def fun(x):
    sleep(x)
    return x

for i in tqdm(range(10)):
    result.append(delayed(fun)(i))

with TqdmCallback(desc="compute"):
    print(compute(result))

screenshot of output in jupyter :