Python multiprocessing performance

Question

This should be my third and final question regarding my attempts to increase performance on some statistical analysis that I am doing with python. I have 2 versions of my code (

Answer 1

Shared data is a known case of slowdowns due to synchronization.

Can you split your data among processes, or give each process an independent copy? Then your processes would not need to synchronize anything up until the moment when all calculations are done.

Then I'd let the master process join the output of all worker processors into one coherent set.

The approach may take extra RAM, but RAM is cheap nowadays.

If you ask, I'm also puzzled by 3700 ms per thread lock acquisition. OTOH profiling may be mistaken about special calls like this.

Answer 2

Your Pastebins is empty.

The problem is that multiprocessing uses fork if its available (instead of spawning a new python proccess). Forked process share same env(file descriptors for example). May be it has some locks among them.

Here is some frustration about that: Multiprocessing or os.fork, os.exec?

Answer 3

As far as the last part of your question, the Python docs basically say that multiprocessing.lock is a clone of threading.lock. Acquire calls on locks can take a long time because if the lock is already acquired, it will block until the lock is released. This can become a problem when multiple processes are competing for access to the same data, like in your code. Because I can't view your pastebin, I can only guess as to exactly what's going on, but most likely, you're processes are acquiring the lock for long periods of time which stops other processes from running, even if there is plenty of free CPU time. This shouldn't be affected by the GIL as that should only constrain multithreaded applications, not multiprocessed ones. So, how to fix this? My guess is that you have some sort of lock protecting your shared array that is staying locked while the process is doing intensive calculations that take a relatively long time, therefore barring access for other processes, which are subsequently blocking on their lock.acquire() calls. Assuming you have enough RAM, I strongly endorse the answer that suggests storing multiple copies of the array in each process's address space. However, just note that passing large data structures through map can cause unexpected bottlenecks, as it requires picking and depickling.