Push a pure-python module to Dask workers

Question

Is there an easy way in Dask to push a pure-python module to the workers?

I have many workers in a cluster and I want to distribute a local module that I have on my cl

Answer 1

Alternative if you wish to deploy a package to the workers after they have started you can do something similar to this using Client.run and Client.restart

def deploy_env(packages):
    conda_prefix = pathlib.Path(sys.executable).parent.parent
    res = subprocess.check_output(['conda', 'install', '-p', conda_prefix] + packages)
    return res

# Run the deploy command on all the workers
result = client.run(deploy_env, packages)

# Restart all the worker processes
client.restart()

After this the packages specified will be installed on all currently running workers.

This approach will not work when adding additional workers to the scheduler.

Answer 2

Yes, use the Client.upload_file method.

client.upload_file('myfile.py')

This method will distribute the file and, if the file ends in .py or .egg will also import and reload the module on each of the workers.