python multiprocessing.Pool kill *specific* long running or hung process

Question

I need to execute a pool of many parallel database connections and queries. I would like to use a multiprocessing.Pool or concurrent.futures ProcessPoolExecutor. Python 2.7.

Answer 1

In your solution you're tampering internal variables of the pool itself. The pool is relying on 3 different threads in order to correctly operate, it is not safe to intervene in their internal variables without being really aware of what you're doing.

There's not a clean way to stop timing out processes in the standard Python Pools, but there are alternative implementations which expose such feature.

You can take a look at the following libraries:

pebble

billiard