combining tqdm with delayed execution with dask in python

问题

tqdm and dask are both amazing packages for iterations in python. While tqdm implements the needed progress bar, dask implements the multi-thread platform and they both can make iteration process less frustrating. Yet - I'm having troubles to combine them both together.

For example, the following code implements a delayed execution in dask, with tqdm.trange progress bar. The thing is that since the delayed is performed quickly, the progress bar ends immediately, while the real computation-time effort is done during the compute part.

from dask import delayed,compute
from tqdm import trange
from time import sleep

ct = time()
result= []

def fun(x):
    sleep(x)
    return x

for i in trange(10):
    result.append(delayed(fun)(i))

print compute(result)

How can I attach the progress bar to the actual execution in compute command?

回答1:

Consider Dask's progress bar

from dask.diagnostics import ProgressBar

with ProgressBar():
    compute(result)

Build a diagnostic of your own

You can use this plugin architecture to get a signal at the end of every task. http://dask.pydata.org/en/latest/diagnostics.html

Here is an example of someone doing exactly this: https://github.com/tqdm/tqdm/issues/278

来源：https://stackoverflow.com/questions/44483950/combining-tqdm-with-delayed-execution-with-dask-in-python