How to deal with “The template parameters are invalid” when launching a custom template using Cloud Dataflow REST API/Python

问题

I have been using Dataprep to build a Dataflow template.

Running it from https://console.cloud.google.com/dataflow/createjob - no problems. It prompts for parameters (regional endpoint, input locations, output locations, custom location for temp files) and the metadata file basically hands me the answers.

When I come to run the custom template from Python using the REST API, I am including the parameters like the below (lots of quote escaping)

BODY = {
    "jobName": "{jobname}".format(jobname=JOBNAME),
    "parameters": {
        "customGcsTempLocation": "gs://{bucket}/dts/temp".format(bucket=BUCKET),
        "inputLocations":"{{\"location1\": \"projectname:DATASET.table\",\"location2\": \"projectname:DATASET.table\",\"location3\": \"projectname:DATASET.table\",\"location4\": \"gs://bucket/filename.csv\",\"location5\":\"projectname:DATASET.table\",\"location6\":\"projectname:DATASET.table\",\"location7\":\"projectname:DATASET.table\",\"location8\":\"projectname:DATASET.table\",\"location9\": [\"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename\"]}}",
        "outputLocations":"{{\"projectname:DATASET.table\"}}"
    },
     "environment": {
        "tempLocation": "gs://{bucket}/dts/temp".format(bucket=BUCKET)
     }
}

The error I get back is "The template parameters are invalid", but there is no more detail; I am not sure if I am missing something, including something I shouldn't, or escaping something incorrectly (other questions like this seem to indicate the latter isn't uncommon). Any advice appreciated, thanks

回答1:

I have been literally experiencing the same issue for the past 6-7 hours, and finally got this to work. It seems that the parameter that is causing this issue is the 'tempLocation'. Also it looks like you have an additional set of {} within your input and output location paramaters. Try the following:

BODY = {
    "jobName": "{jobname}".format(jobname=JOBNAME),
    "parameters": {
        "customGcsTempLocation": "gs://{bucket}/dts/temp".format(bucket=BUCKET),
        "inputLocations":"{\"location1\": \"projectname:DATASET.table\",\"location2\": \"projectname:DATASET.table\",\"location3\": \"projectname:DATASET.table\",\"location4\": \"gs://bucket/filename.csv\",\"location5\":\"projectname:DATASET.table\",\"location6\":\"projectname:DATASET.table\",\"location7\":\"projectname:DATASET.table\",\"location8\":\"projectname:DATASET.table\",\"location9\": [\"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename\"]}",
        "outputLocations":"{\"projectname:DATASET.table\"}"
    }
}

Let me know if that works for you!

回答2:

Cristian was correct but there was one other error I had made, posting in case it helps others in similar situations.

I should have excaped the filenames in location9 differently because they are in a list (rather than just being a value) apparently. The below shows how this was done:

\"location9\":\"[\\\"gs://bucket/folder/folder.tsv\\\",\\\"gs://bucket/folder/filename1122.tsv\\\",\\\"gs://bucket/folder/filename1123.tsv\\\",\\\"gs://bucket/folder/filename1124.tsv\\\",\\\"gs://bucket/folder/filename1125.tsv\\\",\\\"gs://bucket/folder/filename1126.tsv\\\",\\\"gs://bucket/folder/filename1127.tsv\\\",\\\"gs://bucket/folder/filename1128.tsv\\\",\\\"gs://bucket/folder/filename1129.tsv\\\",\\\"gs://bucket/folder/filename1201.tsv\\\",\\\"gs://bucket/folder/filename1202.tsv\\\",\\\"gs://bucket/folder/filename1203.tsv\\\",\\\"gs://bucket/folder/filename1204.tsv\\\",\\\"gs://bucket/folder/filename1205.tsv\\\",\\\"gs://bucket/folder/filename1206.tsv\\\",\\\"gs://bucket/folder/filename1207.tsv\\\",\\\"gs://bucket/folder/filename1208.tsv\\\",\\\"gs://bucket/folder/filename1209.tsv\\\",\\\"gs://bucket/folder/filename1210.tsv\\\",\\\"gs://bucket/folder/filename1211.tsv\\\",\\\"gs://bucket/folder/filename1212.tsv\\\"]\"

来源：https://stackoverflow.com/questions/53778309/how-to-deal-with-the-template-parameters-are-invalid-when-launching-a-custom-t

标签

google-cloud-dataflow