How to add a validation in azure data factory pipeline to check file size?

感情迁移 提交于 2021-02-08 11:49:17

问题


I have multiple data sources I want to add a validation in azure data factory before loading into tables it should check for file size so that it is not empty. So if the file size is more than 10 kb or if it is not empty loading should start and if it is empty then loading should not start. I checked validation activity in Azure Data Factory but it is not showing size for multiple files in a folder. Any suggestions appreciated basically if I can add any python notebook for this validation will also do.


回答1:


Use GetMetadata under General Activities, then send the result to an If Condition.

You will then need to get the file size from the Dataset.@item().name is the name of the file you want to get the size of.

If you are working with a directory do the following:

Then check the file size of each file.

This is what the ForEach settings looks like. Then you can use @item().name inside the ForEach to get at the file.

The data source will need to have the parameter FileName.




回答2:


Following GIF shows step by step process on how to achieve the above requirement in ADF.



来源:https://stackoverflow.com/questions/64124963/how-to-add-a-validation-in-azure-data-factory-pipeline-to-check-file-size

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!