Azure Data Factory V2 Copy Activity - Save List of All Copied Files

问题

I have pipelines that copy files from on-premises to different sinks, such as on-premises and SFTP. I would like to save a list of all files that were copied in each run for reporting.

I tried using Get Metadata and For Each, but not sure how to save the output to a flat file or even a database table.

Alternatively, is it possible to fine the list of object that are copied somewhere in the Data Factory logs?

Thank you

回答1:

Update:

Items:@activity('Get Metadata1').output.childItems

---

---

If you want record the source file names, yes we can. As you said we need to use Get Metadata and For Each activity.
I've created a test to save the source file names of the Copy activity into a SQL table.

1. As we all know, we can get the file list via Child items in Get metadata activity. The dataset of Get Metadata1 activity specify the container which contains several files. The list of file in test container is as follows:
   

2. At inside of the ForEach activity, we can traverse this array. I set a Copy activity named Copy-Files to copy files from source to destnation.

3. @item().name represents every file in the test container. I key in the dynamic content @item().name to specify the file name. Then it will sequentially pass the file names in the test container. This is to execute the copy task in batches, each batch will pass in a file name to be copied. So that we can record each file name into the database table later.

4. Then I set another Copy activity to save the file names into a SQL table. Here I'm using Azure SQL and I've created a simple table.

create table dbo.File_Names(
    Copy_File_Name varchar(max)
);

5. As this post also said, we can use similar syntax select '@{item().name}' as Copy_File_Name to access some activity datas in ADF. Note: the alias name should be the same as the column name in SQL table.

6. Then we can sink the file names into the SQL table.
   Select the table which created previously.
   

7. After I run debug, I can see all the file names are saved into the table.

If you want add more infomation, you can reference the post I maintioned previously.

来源：https://stackoverflow.com/questions/65161230/azure-data-factory-v2-copy-activity-save-list-of-all-copied-files