multiprocessing.Pool seems to work in Windows but not in ubuntu?

Question

SOLVED: The problem was Wingware Python IDE. I guess the natural question now is how it is possible and how this could be fixed.

I asked a question yesterday ( Probl

Answer 1

Update: Turns out this had nothing to do with matplotlib or the backends but rather with a bug associated with multiprocessing in general. We've fixed this for Wing version 4.0.4+. The work-around is not to set breakpoints in the code that is executed in the sub-processes.

It seems to be Wing IDE's matplotlib support for the Tkinter backend interacting badly with multiprocessing. When I try this example it crashes in TCL/Tk code. I suspect the person working on Windows was using a different matplotlib backend.

Turning off the "matplotlib event loop support" in Project Properties under the Extensions tab seems to work around it.

Or, adding the following seems to fix it for me when the "matplotlib event loop support" is turned on.

import matplotlib matplotlib.use('WXAgg')

This will only work if you have the WXAgg backend. Other backends supported by Wing IDE (in such a way that plots remain interactive even if the debug process is paused) are GTKAgg and Qt4Agg but I didn't try those yet.

I'll see if I can find and fix the bug. I suspect we need to disable our event loop support when the process ID changes. Thanks for reporting this.

Answer 2

I don't have a solution for your first problem. In fact, I can run your code repeatedly without fail on my 64-bit Ubuntu box with Enthought's Python 2.7.1 [EPD 7.0-2 (64-bit)]. edit: It turns out the problem was being caused by your IDE (Wingware). The obvious workaround is to run the script from outside the IDE.

As to the second question, what happens is that on Unix every worker process inherits the same state of the random number generator from the parent process. This is why they generate identical pseudo-random sequences. All you have to do to fix this is call scipy.random.seed at the top of testfunc:

def testfunc(x0, N):
    sp.random.seed()
    print 'working with x0 = %s' % x0
    ...