Python multiprocessing: How to close the multiprocessing pool on exception

时光毁灭记忆、已成空白 提交于 2020-01-03 10:47:16

问题


I am using python multiprocessing to split one of the longer processes and run parallelly. It is working fine except when there is an exception in one of the child processes, in which case, process pool is not closed and I can still see those processes on the server.

Here is the code:

from multiprocessing import Pool
pool = Pool(processes=4)
from functools import partial
param_data = "Test Value"
func = partial(test_function, param_data)
r = pool.map(func, range(3))
pool.close()

def test_function(param_data,index):
   try:
      # The process here;
   except Exception as e:
      # Close the process pool;

On giving pool.close inside the except block it says

NameError: global name 'pool' is not defined

I tried to kill the process on Exception with the following code.

    except Exception as e:
       import os
       import signal
       pid = os.getpid()
       os.kill(pid, signal.SIGTERM) 

But I can still see the process on the server. This still not the best solution as this will only terminate the child process which encountered an exception other processes will still go on.

I want all processes to terminate on completion, irrespective of if they encounter with an exception or not.

I am running Python2.7

Ps: I cannot install a new library like psutil on the server, I am trying for a solution using standard python library.

I checked few similar questions such as, auto kill process and child in this forum but they were not really this issue.


回答1:


I got the solution for this - Catch exception at the parent process.

try:
   pool = Pool(processes=4)
   from functools import partial
   param_data = "Test Value"
   func = partial(test_function, param_data)
   r = pool.map(func, range(3))
except Exception as e:
   pool.close()
pool.close()

And also add a try/catch in the child process function:

def test_function(param_data,index):
    try:
       # The process here;
    except Exception as e:
       raise Exception(e.message)          

The except block here looks redundant but it is not. The reason is, some of the exceptions raised by child process were not thrown to the parent process.

For example, I was raising a custom exception in the function and it was not thrown to the parent process, So it is advised to catch all exceptions within the child process and raise the standard Exception from there, Then handle it at parent process, where you can close the process pool or do other cleanups.




回答2:


You have to use: del Pool after Pool.terminate & Pool.join

This will solve your problem



来源:https://stackoverflow.com/questions/44587669/python-multiprocessing-how-to-close-the-multiprocessing-pool-on-exception

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!