How to assign tasks to specific worker within Dask.Distributed

半腔热情 提交于 2021-02-07 07:52:46

问题


I am interesting in using Dask Distributed as task executor. In Celery it is possible to assign task to specific worker. How is it possible using Dask Distributed?


回答1:


There are 2 options:

  1. Specify workers by name or host or IP (but only positive declarations):

    dask-worker scheduler_address:8786 --name worker_1
    

    and then one of option:

    client.map(func, sequence, workers='worker_1')
    client.map(func, sequence, workers=['192.168.1.100', '192.168.1.100:8989', 'alice', 'alice:8989'])
    client.submit(f, x, workers='127.0.0.1')
    client.submit(f, x, workers='127.0.0.1:55852')
    client.submit(f, x, workers=['192.168.1.101', '192.168.1.100'])
    future = client.compute(z, workers={z: '127.0.0.1',
                                    x: '192.168.0.1:9999'})
    future = client.compute(z, workers={(x, y): ['192.168.1.100', '192.168.1.101:9999']})
    
  2. Use Resources concept. You can specify available resources to worker like:

    dask-worker scheduler:8786 --resources "CAN_PROCESS_QUEUE_ALICE=2"
    

    and specify required resources like

    client.submit(aggregate, processed, resources={'CAN_PROCESS_QUEUE_ALICE': 1})
    

    or

    z = some_dask_object.map_parititons(func)
    z.compute(resources={tuple(y.__dask_keys__()): {'CAN_PROCESS_QUEUE_ALICE': 1})
    


来源:https://stackoverflow.com/questions/51479536/how-to-assign-tasks-to-specific-worker-within-dask-distributed

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!