Multiprocessing working in Python but not in iPython

Question

I am running the following code in iPython:

import multiprocessing

def my_function(x):
    \"\"\"The function you want to compute in parallel.\"\"\"
    x +         


        

Answer 1

At least in the current version of Jupyter Notebook (the successor of IPython) you can solve this by moving the target function to a separate module, and importing it.

I have no idea why this works, it's rather odd, but it does.

i.e. - in a workers.py file put

def my_function(x):
    """The function you want to compute in parallel."""
    x += 1
    return x

Then in IPython/Jupyter notebook put:

import multiprocessing
import workers

pool = multiprocessing.Pool()
results = pool.map(workers.my_function, [1,2,3,4,5,6])
print(results)

also - the if-main thingy doesn't seem to be needed.

Credit: Gaurav Singhal

Answer 2

From the documentation:

Note

Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter