azure-data-factory

问题 My dev ADF has pipeline triggers that are Active and set to run every hour. Using devops repos & CI/CD, I deployed these triggers from our dev to test ADF environments. The pipeline triggers were successfully created in my test environment, but they are inactive. Why are these not coming over in a 'Started' status. The ARM template shows the trigger as Started These need to be turned on automatically after deployment as I don't have rights to turn these on manually. I can't tell if this is a

问题 I am attempting a simple SELECT action on a Source JSON dataset in an Azure Data Factory data flow, but I am getting an error message that none of the columns from my source are valid. I use the exact configuration as the video, except instead of a CSV file, I use a JSON file. In the video, at 1:12, you can see that after configuring the source dataset, the source projection shows all of the columns from the source schema. Below is a screen shot from the tutorial video: image.png And below is

问题 ADF git mode recommends publishing to Master branch(or a fix branch which usually is protected from check-ins). Since Master is usually protected from check-ins and only pull requests are allowed, how would the publish work? 回答1: Master branch is used to publish ADF to adf_publish branch. In short, this process converts several json files into ARM-template json files stored in adf_publish branch. Other working banches need to be merged into Master branch and then be published together in

问题 Azure data factory introduces many branches in git mode. Source branch : Save is stored here Publish branch working branch: After a git is connected, ADF prompts us for this. What does the working branch do? 回答1: I think there are three types of branches when we use git mode. Branch Name Quantitiy Use Range adf_publish One and only one save Resource Manager templates of the published factory main One and only one this is collaboration branch used to merge changes of other working branches

问题 I download json files from a web API and store them in blob storage using a Copy Data activity and binary copy. Next I would like to use another Copy Data activity to extract a value from each json file in the blob container and store the value together with its ID in a database. The ID is part of the filename, but is there some way to extract the filename? 回答1: You can do the following set of activities: 1) A GetMetadata activity, configure a dataset pointing to the blob folder, and add the

问题 I have pipelines that copy files from on-premises to different sinks, such as on-premises and SFTP. I would like to save a list of all files that were copied in each run for reporting. I tried using Get Metadata and For Each, but not sure how to save the output to a flat file or even a database table. Alternatively, is it possible to fine the list of object that are copied somewhere in the Data Factory logs? Thank you 回答1: Update: Items: @activity('Get Metadata1').output.childItems If you

问题 I need to migrate all my data from Azur data lake Gen1 to Lake Gen2. In my lake we have different types of file mixed (.txt, .zip,.json and many other). We want to move them as-it-is to GEN2 lake. Along with that we also want to maintain last updated time for all files as GEN1 lake. I was looking to use ADF for this use case. But for that we need to define dataset, and to define dataset we have to define data format(Avro,json,xml, binary etc). As we have different type of data mixed, I tried

问题 seemingly simple task. I just want to rename a blob file, I know I have to copy it to rename or something, then delete the original but this is proving tricky. I have created the storage context (New-AzureStorageContext), and got the blob (Get-AzureStorageBlob), and found Start-AzureStorageBlobCopy, but how to I actually rename it? I'd like to do this within the same container if possible as well. Ideally I'd run it in an Azure Runbook and call it using a webhook I Azure Data Factory v2. I

问题 I have a new pipeline in azure data factory. I created the dataset, one from the rest api (a public one): https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=MSFT&apikey=demo and then I created an azure sql table with columns shown in the screenshot The problem, is that I dont know how to do the mapping, as this is a complex JSON object, I am limited with the Mapping Designer: How do I map the date? 回答1: I tend to use an ELT approach for these, calling the REST API with a Web

问题 I have filter in Data Flow of Azure Data Factory where I need to write expression for the following SQL logic(Where statement): Where ID NOT IN (Select PID from Table1 where ptype_c in (100, 2000)) 回答1: If your source is SQL Database, you can just choose Query option. If no, you can try like this. 1.create a Filter activity, get the data which ptype_c is in (100, 2000). 2.create a Select activity, change the column name. 3.new a branch of Source. 4.create a Exist activity. 来源： https:/