multiprocessing Pool hangs when there is a exception in any of the thread

问题

I am new to Python and trying a multiprocessing.pool program to process files, it works fine as long as there are no exceptions. If any of the thread/process gets an exception the whole program waits for the thread

snippet of the code:

cp = ConfigParser.ConfigParser()
cp.read(gdbini)
for table in cp.sections():
    jobs.append(table)
#print jobs
poolreturn = pool.map(worker, jobs)
pool.close()
pool.join()

Failure Message:

---

Traceback (most recent call last):
  File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/cnet-python/default-2.6/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'ConfigParser.NoOptionError'>, ("No option 'inputfilename' in section: 'section-1'",))

---

I went ahead added a exception handler to terminate the process

try:
    ifile=cp.get(table,'inputfilename')
except ConfigParser.NoSectionError,ConfigParser.NoOptionError:
    usage("One of Parameter not found for"+ table)
    terminate()

but still it waits, not sure whats missing.

回答1:

In Python 3.2+ this works as expected. For Python 2, this bug was fixed in r74545 and will be available in Python 2.7.3. In the mean time, you can use the configparser library which is a backport of the configparser from 3.2+. Check it out.

回答2:

I had the same issue. It happens when a worker process raises a user exception which has a custom constructor. Make sure your exception (ConfigParser.NoOptionError in that case) initializes the base exception with exactly two arguments:

class NoOptionError(ValueError):

    def __init__(self, message, *args):
        super(NoOptionError, self).__init__(message, args)

来源：https://stackoverflow.com/questions/2246384/multiprocessing-pool-hangs-when-there-is-a-exception-in-any-of-the-thread