dask-kubernetes

问题 I have some long-running code (~5-10 minute processing) that I'm trying to run as a Dask Future . It's a series of several discrete steps that I can either run as one function: result : Future = client.submit(my_function, arg1, arg2) Or I can split up into intermediate steps: # compose the result from the same intermediate results but with Futures intermediate1 = client.submit(my_function1, arg1) intermediate2 = client.submit(my_function2, arg1, arg2) intermediate3 = client.submit(my

问题 I have some long-running code (~5-10 minute processing) that I'm trying to run as a Dask Future . It's a series of several discrete steps that I can either run as one function: result : Future = client.submit(my_function, arg1, arg2) Or I can split up into intermediate steps: # compose the result from the same intermediate results but with Futures intermediate1 = client.submit(my_function1, arg1) intermediate2 = client.submit(my_function2, arg1, arg2) intermediate3 = client.submit(my

问题 I am learning dask-kubernetes on GKE. I stumbled across an asyncio error (ERROR:asyncio:Task exception was never retrieved). See steps below for the issue. However, additional guidance on using deploying dask-kubernetes with a remote Kubernetes cluster is appreciated (note I used helm with good experience here but want to try the native approach as I can't scale the helm approach). Create the cluster: $ gcloud container clusters create --machine-type n1-standard-2 --num-nodes 2 --zone us