python executor spawn tasks from done callback (recursively submit tasks)

你。 提交于 2021-02-09 01:57:52

问题


I'm trying to submit further tasks from result of a task that was done:

with concurrent.futures.ThreadPoolExecutor() as executor:
    future = executor.submit(my_task)
    def callback(future):
        for another_task in future.result():
            future = executor.submit(another_task)
            future.add_done_callback(callback)
    future.add_done_callback(callback)

but I'm getting:

RuntimeError: cannot schedule new futures after shutdown

What's the best way to make the executor hold for the callback? A semaphore?

Ideally the solution should transfer well if ThreadPoolExecutor is replaced with ProcessPoolExecutor.


回答1:


The callback is executed in a separate thread. Therefore, when your callback logic starts executing, there's a high chance the main loop will leave the context manager shutting down the Executor. That's why you observe the RuntimeError.

The easiest fix is running your logic sequentially.

futures = []

with concurrent.futures.ThreadPoolExecutor() as pool:
    futures.append(pool.submit(task))

    while futures:
        for future in concurrent.futures.as_completed(futures):
            futures.remove(future)

            for new_task in future.result():
                futures.append(pool.submit(new_task))

Note that this code might cause an exponential submission of tasks to the Executor.




回答2:


This solution is guaranteed to max out processing at any given point. The use of sleep is not that elegant but this is the best I have so far.

with concurrent.futures.ThreadPoolExecutor() as executor:
    pending_tasks = 1
    future = executor.submit(my_task)
    def callback(future):
        nonlocal pending_tasks
        for another_task in future.result():
            pending_tasks += 1
            future = executor.submit(another_task)
            future.add_done_callback(callback)
        pending_tasks -= 1
    future.add_done_callback(callback)
    while pending_tasks:
        time.sleep(10)


来源:https://stackoverflow.com/questions/51879070/python-executor-spawn-tasks-from-done-callback-recursively-submit-tasks

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!