Import custom modules on IPython.parallel engines with sync_imports()

Question

I\'ve been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven\'t been able to do it as explained on the cookbook using dview

Answer 1

The problem is that you're changing the PYTHONPATH just in the local process running the Client, and not in the remote processes running in the ipcluster.

You can observe this behaviour if you run the next piece of code:

from IPython.parallel import Client

rc = Client()
dview = rc[:]

with dview.sync_imports():
    import sys
    sys.path[:] = ['something']
   
def parallel(x):
    import sys
    return sys.path

print 'Local: ', sys.path
print 'Remote: ', dview.map_sync(parallel, range(1))

Basically all the modules that you want to use with sync_imports must already be in the PYTHONPATH.

If it's not in the PYTHONPATH then you must add it to the path in the function that you execute remotely, and then import the module in the function.