Azure datafactory v2 Execute Pipeline with For Each

末鹿安然 提交于 2021-01-29 19:57:48

问题


I am trying to use "Execute Pipeline" to invoke a Pipe which has a ForEach activity. I get an error.

  1. Json for Execute pipe:
[
    {
        "name": "pipeline3",
        "properties": {
            "activities": [
                {
                    "name": "Test_invoke1",
                    "type": "ExecutePipeline",
                    "dependsOn": [],
                    "userProperties": [],
                    "typeProperties": {
                        "pipeline": {
                            "referenceName": "MAIN_SA_copy1",
                            "type": "PipelineReference"
                        },
                        "waitOnCompletion": true
                    }
                }
            ],
            "annotations": []
        }
    }
]
  1. Jason for Invoke pipe for each activity :
[
    {
        "name": "MAIN_SA_copy1",
        "properties": {
            "activities": [
                {
                    "name": "Collect_SA_Data",
                    "type": "ForEach",
                    "dependsOn": [],
                    "userProperties": [],
                    "typeProperties": {
                        "items": {
                            "value": "@pipeline().parameters.TableNames",
                            "type": "Expression"
                        },
                        "batchCount": 15,
                        "activities": [
                            {
                                "name": "Sink_SAdata_toDL",
                                "type": "Copy",
                                "dependsOn": [],
                                "policy": {
                                    "timeout": "7.00:00:00",
                                    "retry": 0,
                                    "retryIntervalInSeconds": 30,
                                    "secureOutput": false,
                                    "secureInput": false
                                },
                                "userProperties": [
                                    {
                                        "name": "Destination",
                                        "value": "@{pipeline().parameters.DLFilePath}/@{item()}"
                                    }
                                ],
                                "typeProperties": {
                                    "source": {
                                        "type": "SqlServerSource",
                                        "sqlReaderQuery": {
                                            "value": "@concat('SELECT * FROM ',item())",
                                            "type": "Expression"
                                        }
                                    },
                                    "sink": {
                                        "type": "AzureBlobFSSink"
                                    },
                                    "enableStaging": false,
                                    "parallelCopies": 1,
                                    "dataIntegrationUnits": 4
                                },
                                "inputs": [
                                    {
                                        "referenceName": "SrcDS_StructuringAnalytics",
                                        "type": "DatasetReference"
                                    }
                                ],
                                "outputs": [
                                    {
                                        "referenceName": "ADLS",
                                        "type": "DatasetReference",
                                        "parameters": {
                                            "FilePath": "@pipeline().parameters.DLFilePath",
                                            "FileName": {
                                                "value": "@concat(item(),'.orc')",
                                                "type": "Expression"
                                            }
                                        }
                                    }
                                ]
                            }
                        ]
                    }
                }
            ],
            "parameters": {
                "DLFilePath": {
                    "type": "string",
                    "defaultValue": "extracts/StructuringAnalytics"
                },
                "TableNames": {
                    "type": "array",
                    "defaultValue": [
                        "fom.FOMLineItem_manual"
                    ]
                }
            },
            "variables": {
                "QryTableColumn": {
                    "type": "String"
                },
                "QryTable": {
                    "type": "String"
                }
            },
            "folder": {
                "name": "StructuringAnalytics"
            },
            "annotations": []
        },
        "type": "Microsoft.DataFactory/factories/pipelines"
    }
]

I get an error:

[
    {
        "errorCode": "BadRequest",
        "message": "Operation on target Collect_SA_Data failed: The execution of template action 'Collect_SA_Data' failed: the result of the evaluation of 'foreach' expression '@pipeline().parameters.TableNames' is of type 'String'. The result must be a valid array.",
        "failureType": "UserError",
        "target": "Test_invoke1",
        "details": ""
    }
]

Input:

"pipeline": {
    "referenceName": "MAIN_SA_copy1",
    "type": "PipelineReference"
},
"waitOnCompletion": true,
"parameters": {
    "DLFilePath": "extracts/StructuringAnalytics",
    "TableNames": "[\"fom.FOMLineItem_manual\"]"
}

回答1:


Please try updating your dynamic expression of ForEach Items as below:

{
    "value": "@array(pipeline().parameters.TableNames)",
    "type": "Expression"
}

Hope this helps.




回答2:


I guess you were using the UI to set the pipeline and its parameters and I guess you expected to put the array parameter of the called pipeline as everywhere else like this: (It is all my guess, because I just did exactly the same, with the same result)

The trick is to define the array in the Code (["table1", "table2"]):

The input in the UI will look like this:

Now it works!
It seems, that the Datafactory is otherwise treating the whole array as one element of some array. Hence, the solution with the array() function sometimes works.
It looks like a bug, defining array parameter input..

(Had to edit the answer, I first thought omiting the colons in the UI input would be enough)



来源:https://stackoverflow.com/questions/61109985/azure-datafactory-v2-execute-pipeline-with-for-each

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!