How can I provide runtime parameter/argument values via a file for CDAP pipleline

Question

How do i add run time parameter/arguments to CDAP pipeline.

We can set parameters in all the Transforms, Source and Target nodes and manually enter the values at run tim

Answer 1

When starting a pipeline, you can specify runtime arguments as a JSON map in the request body.

First, your pipeline should contain property lookup macros, which are documented here. As an example, a pipeline json definition could contain the following configuration for the BigQuery Sink plugin (surrounding fields removed for brevity):

  {
    "name": "BigQuerySink",
    "plugin": {
      "properties": {
        "table": "${bqsink_table}"
      }
    },

Then, when starting your pipeline, you can specify the value(s) for the macros, as mentioned here.

In this example, you would send a POST request to v3/namespaces/<namespace-id>/apps/<app-id>/workflows/DataPipelineWorkflow/start, with request body:

{ "bqsink_table": "myTable" }