问题
How do i add run time parameter/arguments to CDAP pipeline.
We can set parameters in all the Transforms, Source and Target nodes and manually enter the values at run time in the Datafusion environments.
However in a production environments i would want a parameter file to feed the values to the pipeline, how do i achieve this functionality.
In the CDAP documentation i don't find anything useful regarding the run time arguments. I am new to datafusion and am not very familiar with the interface. If this question is already answered a redirect to the thread will also be helpful
回答1:
When starting a pipeline, you can specify runtime arguments as a JSON map in the request body.
First, your pipeline should contain property lookup macros, which are documented here. As an example, a pipeline json definition could contain the following configuration for the BigQuery Sink plugin (surrounding fields removed for brevity):
{
"name": "BigQuerySink",
"plugin": {
"properties": {
"table": "${bqsink_table}"
}
},
Then, when starting your pipeline, you can specify the value(s) for the macros, as mentioned here.
In this example, you would send a POST request to v3/namespaces/<namespace-id>/apps/<app-id>/workflows/DataPipelineWorkflow/start
, with request body:
{ "bqsink_table": "myTable" }
来源:https://stackoverflow.com/questions/59677435/how-can-i-provide-runtime-parameter-argument-values-via-a-file-for-cdap-piplelin