azure-data-factory

问题 I have my data stored in blobs and I have written a python script to do some computations and create another csv. How can I execute this in Azure Data Factory ? 回答1: Mighty. You could use Azure Data Factory V2 custom activity for your requirements. You can directly execute a command to invoke python script using Custom Activity. Please refer to this sample on the github. Hope it helps you. 回答2: Another option is using a DatabricksSparkPython Activity. This makes sense if you want to scale out

问题 I'm looking for a way to be able to connect to Databricks deltalake tables from ADF and other Azure Services(like Data Catalog). I don't see databricks data store listed in ADF data sources. On a similar question - Is possible to read an Azure Databricks table from Azure Data Factory? @simon_dmorias seems to have suggested using ODBC connection to connect to databricks tables. I tried to set up the ODBC connection but it requires IR to be setup. There are 2 options I see when creating the IR.

问题 I have created a copy activity that copies data from an on premise database to a Azure SQL Database. I need to modify dynamiccaly the query so it take a range of date, so I create two variables: - inidate - enddate that I want to use inside the where clause, but I don't know how to reference the variables. I tried this but it doesn't work: "SELECT * FROM tableOnPrem WHERE dateOnPrem BETWEEN '@variable('inidate')' AND '@variable('enddate')' please help. meny thanks 回答1: In a Pipeline (like for

问题 I have an Azure data factory v2 pipeline containing an until activity. Inside the until is a copy activity - if this fails, the error is logged, exactly as in this post, and I want the loop to continue. Azure Data Factory Pipeline 'On Failure' Although the inner copy activity’s error is handled, the until activity is deemed to have failed because an inner activity has failed. Is there any way to configure the until activity to continue when an inner activity fails? 回答1: Solution Put the error

问题 I am using ADF to execute Databricks notebook. At this time, I have 6 pipelines, and they are executed consequently. Specifically, after the former is done, the latter is executed with multiple parameters by the loop box, and this keeps going. For example, after the first pipeline is done, it will trigger 3 instances of the second pipeline with different parameters, and each of these instances will trigger multiple instances of the third pipeline. As a result, the deeper I go, the more

问题 I am using ADF to execute Databricks notebook. At this time, I have 6 pipelines, and they are executed consequently. Specifically, after the former is done, the latter is executed with multiple parameters by the loop box, and this keeps going. For example, after the first pipeline is done, it will trigger 3 instances of the second pipeline with different parameters, and each of these instances will trigger multiple instances of the third pipeline. As a result, the deeper I go, the more

问题 I would like to execute a REST API with oauth authentication using ADF Pipeline. Without oauth2, I could call any REST APIs. So here my question is, does this ADF pipeline support REST APis with oauth2 authentication ? if yes. Please provide a solution The API which is am trying, you can find from this URL https://docs.microsoft.com/en-us/rest/api/datafactory/pipelineruns/querybyfactory Thanks Binu 回答1: You could call the REST API with a Web activity in the pipeline, select the Authentication

问题 I was trying to invoke data factory pipeline from azure function programmatically. Its throwing following error. link: http://eatcodelive.com/2016/02/24/starting-an-azure-data-factory-pipeline-from-c-net/ AuthorizationFailed: The client 'XXXX-XXXXX-XXXX' with object id 'XXX829e05'XXXX-XXXXX' does not have authorization to perform action 'Microsoft.DataFactory/datafactories/datapipelines/read' over scope '/subscriptions/XXXXXX-4bf5-84c6-3a352XXXXXX/resourcegroups/fffsrg/providers/Microsoft

问题 I have an ADF pipeline that uses copy data activity for copying data from blob storage to table storage. This pipeline runs on a trigger once every day. I have provided a table name in table storage data set as 'Table1'. Instead of providing a hard coded table name value (Table1), is it possible to provide a dynamic value as table name in the table storage such that RUN ID of pipeline run is used as the table name in the table storage and copy data from blob to that table in table storage?

问题 Using azure data factory v2 with GIT / Azure DevOps integration: If you for example create a trigger using Set-AzDataFactoryV2Trigger via powershell according to the documentation, the trigger is created directly in the adf_publish branch. This is an issue, as this will result in a mismatch between the master branch and adf_publish, meaning you'll not be able to publish going forward as this of course raises an error. How do I get the cmdlet to create the trigger in a new or specific branch,