问题
I am using ADF to execute Databricks notebook. At this time, I have 6 pipelines, and they are executed consequently.
Specifically, after the former is done, the latter is executed with multiple parameters by the loop box, and this keeps going. For example, after the first pipeline is done, it will trigger 3 instances of the second pipeline with different parameters, and each of these instances will trigger multiple instances of the third pipeline. As a result, the deeper I go, the more pipelines I have to run.
The issue with me is: when each pipeline is executed, it will ask Databricks to allocate a cluster to run. However, Databricks limits the number of cores to be used for each workspace, which causes the pipeline instance to fail to run.
My question is: is there any solution to control the number of pipeline instance running at the same time, or any solution to handle my issue?
Thanks in advance :-)
回答1:
Why this issue occurs?
Note: Creating a Databricks clusters always have a dependency with the number of cores available in the subscription.
Before creating any databricks cluster, make sure number of cores are available in the region selected and the VM Family vCPUs.
You can checkout the core limit of your subscription by going to Azure Portal => Subscriptions => Select your subscription => Settings “Usage + quotes” => Checkout the usage quota available for each regions.
Example: If your subscription has > 72 cores which results in success of ADF runs else results in failure.
Activity Validate failed: Databricks execution failed with error message: Unexpected failure while waiting for the cluster to be ready. Cause Unexpected state for cluster (job-200-run-1): Could not launch cluster due to cloud provider failures. azure_error_code: OperationNotAllowed, azure_error_message: Operation results in exceeding quota limits of Core. Maximum allowed: 350, Current in use: 344
I’m trying to create 6 pipelines with databricks clusters with 2 worker nodes each. Which means it requires
(6 pipelines) * (1 Driver Node + 2 Worker Node) * (4 cores) = 72 cores.
Above calculation used with VM Size Standard_DS3_v2 which has 4 cores.
Note: To create a databricks spark cluster which requires more than 4 cores i.e. (Minimum 4 cores for Driver type and 4 cores for Worker type).
Resolutions for this issue:
- Increase the core limits by raising the ticket to billing and subscription team to higher limit. With this option once you use it you will be charged for the used cores only.
- Limit your job frequency so that limited number of clusters/ consider using a single job for copying multiple file so that you can limit the cluster creations which will exhaust your cores under your subscription.
To request an increase of one or more resources that support such an increase, submit an Azure support request (select "Quota" for Issue type).
Issue type: Service and subscription limits (quotas)
Reference: Total regional vCPU limit increases
Hope this helps. Do let us know if you any further queries.
Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.
回答2:
You can limit number of activities being run in parallel at each foreach level by setting - Batch Count Parameter. ( Found under settings tab on foreach loop)
batchCount- Batch count to be used for controlling the number of parallel execution (when isSequential is set to false).
https://docs.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity
If not able to set limit at overall Pipeline level, try arriving at a minimum value of batch count in your each nested foreach loops.
来源：https://stackoverflow.com/questions/59292922/azure-data-factory-limit-the-number-of-databricks-pipeline-running-at-the-same