How can I provide runtime parameter/argument values via a file for CDAP pipleline

只谈情不闲聊 提交于 2021-02-05 09:39:38

问题


How do i add run time parameter/arguments to CDAP pipeline.

We can set parameters in all the Transforms, Source and Target nodes and manually enter the values at run time in the Datafusion environments.

However in a production environments i would want a parameter file to feed the values to the pipeline, how do i achieve this functionality.

In the CDAP documentation i don't find anything useful regarding the run time arguments. I am new to datafusion and am not very familiar with the interface. If this question is already answered a redirect to the thread will also be helpful


回答1:


When starting a pipeline, you can specify runtime arguments as a JSON map in the request body.

First, your pipeline should contain property lookup macros, which are documented here. As an example, a pipeline json definition could contain the following configuration for the BigQuery Sink plugin (surrounding fields removed for brevity):

  {
    "name": "BigQuerySink",
    "plugin": {
      "properties": {
        "table": "${bqsink_table}"
      }
    },

Then, when starting your pipeline, you can specify the value(s) for the macros, as mentioned here.

In this example, you would send a POST request to v3/namespaces/<namespace-id>/apps/<app-id>/workflows/DataPipelineWorkflow/start, with request body:

{ "bqsink_table": "myTable" }


来源:https://stackoverflow.com/questions/59677435/how-can-i-provide-runtime-parameter-argument-values-via-a-file-for-cdap-piplelin

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!