azure-data-factory

问题 I am trying to use "Execute Pipeline" to invoke a Pipe which has a ForEach activity. I get an error. Json for Execute pipe: [ { "name": "pipeline3", "properties": { "activities": [ { "name": "Test_invoke1", "type": "ExecutePipeline", "dependsOn": [], "userProperties": [], "typeProperties": { "pipeline": { "referenceName": "MAIN_SA_copy1", "type": "PipelineReference" }, "waitOnCompletion": true } } ], "annotations": [] } } ] Jason for Invoke pipe for each activity : [ { "name": "MAIN_SA_copy1"

问题 We are currently developing a ETL solution in Azure Data Factory that requires calling out to an Azure Function App HTTP trigger. Out Data Factory instance has a Managed Identity configured and I was wondering how I can secure the Azure Function App to only allow access from the Data Factroy Managed Identity? I have previously used Azure Function Apps System Assigned Managed Identities to access other resources (Key Vault) so I understand the basic concepts but I am struggling to understand

问题 Looking for information on how to load data received over email (attached as .xslx or .csv ), and load them into data storage with Azure Data Factory. I've been unable to search for a solution --- all I can find with keywords "Data Factory" and "email" will direct me to questions like "how to set up email notifications for data factory." Any pointers on which pipeline module to use or any related articles are greatly appreciated. Thanks! 回答1: Actually,ADF copy activity supports office 365

问题 I have a Azure data factory pipeline that load data of every working day(Run every working day). I want to trigger my pipeline every working day (Mon-Fry) between working hour(9am to 6pm) and hourly. It should run as daily at 9am then 10am then 11am--------at 6pm. I have tried tumbling window trigger but I think it does not support time period for trigger interval 回答1: The recurrence patterns in ADF and Logic Apps don't directly support this combination of requirements. You are certainly

问题 I am trying to use a Lookup Activity to return a row count. I am able to do this, but once I do, I would like to run an If Statement against it and if the count returns more than 20MIL in rows, I want to execute an additional pipeline for further table manipulation. The issue, however, is that I can not compare the returned value to a static integer. Below is the current Dynamic Expression I have for this If Statement: @greater(int(activity('COUNT_RL_WK_GRBY_LOOKUP').output),20000000) and

问题 We have different ADF environments such as TST,UAT & prod. In each environment the self hosted IR name is different. Is it possible to parameterize the integration runtime in ADF ARM Template, so that when the ARM templates deploys through CI/CD we can pass IR name in the DevOps Variable group during the deployment. I have tried changing the Parametrization Template in ADF with below setting, but the IR name is still not available in the ARM template parameter JSON 'arm_template_parameters

问题 I'm trying to use Azure Data Factory to upsert a CSV into an Azure SQL table. All seemed well until I checked the results. One of the columns is a nullable date. The CSV contains a value like so 1/2/2020 12:00:00 AM . The data flow silently inserts a NULL instead of throwing an error because it didn't like the input. So how can I get my data flow to convert the string to a datetime properly, and then to error out on issues like this in the future? I really don't want silent failures and bad

问题 I have column ABC where timestamp is of format dd/MM/yyyy HH:mm:SS (11/04/2020 1:17:40).I want to create another column ABC_NEW with same data as old column but with different timestamp 'yyyy-MM-dd HH:mm:SS'.I tried doing in azure data factory derived column using toTimestamp(column_name,'yyyy-MM-dd HH:mm:SS') but it did not work it is coming as NULL. Can anyone help? 回答1: It's a 2-step process. You first need to tell ADF what each field in your timestamp column represents, then you can use

问题 Curious to find out what the best way is to generate relationship identities through ADF. Right now, I'm consuming JSON data that does not have any identity information. This data is then transformed into multiple database sink tables with relationships (1..n, etc.). Due to FK constraints on some of the destination sink tables, these relationships need to be "built up" one at a time. This approach seems a bit kludgy, so I'm looking to see if there are other options that I'm not aware of. Note

问题 My dev ADF has pipeline triggers that are Active and set to run every hour. Using devops repos & CI/CD, I deployed these triggers from our dev to test ADF environments. The pipeline triggers were successfully created in my test environment, but they are inactive. Why are these not coming over in a 'Started' status. The ARM template shows the trigger as Started These need to be turned on automatically after deployment as I don't have rights to turn these on manually. I can't tell if this is a