Use Python Pool with context manager or close and join

人盡茶涼 提交于 2021-02-08 12:37:11

问题


The Python documentation has examples in the format of

with Pool() as p:
    p.map(do)

but I see a lot of people using the format below.

p = Pool()
p.map(do)
p.close()
p.join()

Which is more desirable?


回答1:


I think using Pool as a context manager (e.g., with ...) is desirable. It's a newer addition to Pool, and it lets you more cleanly encapsulate the lifespan of the pool.

One thing to be aware of is, that when the context manager exits, it will terminate the pool and any ongoing tasks. This means that you still want to do p.join() in some cases. Your example doesn't require this, because p.map will block execution until the task is done anyway:

A parallel equivalent of the map() built-in function (it supports only one iterable argument though). It blocks until the result is ready.

https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing.pool.Pool.map

Therefore, in the second example, the call to .join() is unnecessary, as .map() will block until all tasks have completed.

However, using .map_async would make .join useful:

with Pool() as p:
    p.map_async(do_something, range(100))
    # Do something else while tasks are running
    p.close()
    p.join()

Edit: as Facundo Olano points out, .close() must always be called before .join(), as stated in the docs:

Wait for the worker processes to exit. One must call close() or terminate() before using join().

https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing.pool.Pool.join



来源:https://stackoverflow.com/questions/55035333/use-python-pool-with-context-manager-or-close-and-join

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!