azure-data-factory

问题 i have a requirement to create an ADF pipeline using ARM template in powershell and it has to take inputs/validate few things from existing ADF piepline, for that reason i have to download the ARM tempalte for existing ADF pipeline through powershell. Can we do that for single ADF pipeline or multiple ones? Note: existing pipeline is not created through ARM deployment, so i cant use "Save-AzureRmDeploymentTemplate" as i dont have deployment name created when pipeline is created through portal

问题 i have a requirement to create an ADF pipeline using ARM template in powershell and it has to take inputs/validate few things from existing ADF piepline, for that reason i have to download the ARM tempalte for existing ADF pipeline through powershell. Can we do that for single ADF pipeline or multiple ones? Note: existing pipeline is not created through ARM deployment, so i cant use "Save-AzureRmDeploymentTemplate" as i dont have deployment name created when pipeline is created through portal

问题 Is there a way to publish ADF pipeline run events with status to App Insights? 回答1: Per my knowledge, you could use Web Activity in the ADF to invoke the Application Insights REST API after execution of your main activities(Or using Execute Pipeline Activity to execute your root pipeline and get the status or output of it).Then send it to App Insights REST API. More details,please refer to this document:https://www.ben-morris.com/using-azure-data-factory-with-the-application-insights-rest-api

问题 Is there a way to publish ADF pipeline run events with status to App Insights? 回答1: Per my knowledge, you could use Web Activity in the ADF to invoke the Application Insights REST API after execution of your main activities(Or using Execute Pipeline Activity to execute your root pipeline and get the status or output of it).Then send it to App Insights REST API. More details,please refer to this document:https://www.ben-morris.com/using-azure-data-factory-with-the-application-insights-rest-api

问题 Calling azure rest api (/consumption/usagedetails) in webactivity using azure data factory .It throws error The length of execution output is over limit (around 1M currently).Any suggestion 回答1: Please reference this blog: Web activity throws overlimit error when calling rest api, Jay Gong has given the answer. Web activity has times out limitation for 1 minute. Also, based on the above error The length of execution ouput is over limit (around 1M currently). , web activity also has output

问题 I have multiple data sources I want to add a validation in azure data factory before loading into tables it should check for file size so that it is not empty. So if the file size is more than 10 kb or if it is not empty loading should start and if it is empty then loading should not start. I checked validation activity in Azure Data Factory but it is not showing size for multiple files in a folder. Any suggestions appreciated basically if I can add any python notebook for this validation

问题 I have linked azure function in data factory pipeline which writes the text file to blob storage The azure function works fine when executed independently and writes the file to blob storage But i am facing below mentioned error when i run the azure function from data factory { "errorCode": "3600", "message": "Error calling the endpoint.", "failureType": "UserError", "target": "Azure Function1" } I have configured the azure fucntion to access the blob with blobendpoint and shared access

问题 Hi I am new to Azure data factory and not all familiar with the back-end processing that run behind the scenes. I am wondering if there is a performance impact to running couple of data flows in parallel when compared to having all the transformations in one data flow. I am trying to stage some data with a not exists transformation. i have to do it for multiple tables. when i test ran two data flows in parallel the clusters were brought up together for both the data flows simultaneously. But

问题 In git mode, when we want to test a pipeline, ADF forces us to publish first. Publish action does two things is my understanding Saves to the local ADF (DEV) as given here Creates arm templates in a branch (adf_publish/the branch we specify) But to get the 'Publish' button enabled, we need to be in collaboration branch. This means no two people can work at the same time on a DEV ADF. As both people will be asked to publish by ADF before they could test the pipeline they are building. If this

问题 In git mode, when we want to test a pipeline, ADF forces us to publish first. Publish action does two things is my understanding Saves to the local ADF (DEV) as given here Creates arm templates in a branch (adf_publish/the branch we specify) But to get the 'Publish' button enabled, we need to be in collaboration branch. This means no two people can work at the same time on a DEV ADF. As both people will be asked to publish by ADF before they could test the pipeline they are building. If this