Python multiprocessing.pool's interaction with a class objective function and neuro-evolution

Question

Warning, this is gonna be long since I want to be as specific as I can be.

---

Exact problem: This is a multi-processing problem. I have ensured

Answer 1

You are not using multiple threads. You are using multiple processes.

All arguments that you pass to apply_async, including the function itself, are serialized (pickled) under the hood and passed to a worker process via an IPC channel (read up multiprocessing documentation for details). So you cannot pass any entities that are tied to things that are by their nature process-local. This includes most synchronization primitives since they have to use locks to do atomic operations.

Whenever this happens (as many other questions on this error message show), you are likely trying to be too smart and passing to a parallelization framework an object that already has parallelization logic built in.

---

If you want to create "multiple levels of parallelization" with such "parallelized object", you'll be better off either:

• using the parallelization mechanism of that object proper and not bother about multiple levels: you can't do more stuff at a time than you have cores anyway; or
• create and use these "parallelized objects" inside worker processes
  • but you are likely to hit multiprocessing limitations here since its worker processes are deliberately prohibited from spawning their own pools.
    • You can let workers add extra items to the work queue but may hit Queue limitations as well.
  • so for such a scenario, a more advanced 3rd-party distributed work queue solution may be preferrable.