How to deal with “The template parameters are invalid” when launching a custom template using Cloud Dataflow REST API/Python

Question

I have been using Dataprep to build a Dataflow template.

Running it from https://console.cloud.google.com/dataflow/createjob - no problems. It prompts for parameters (re

Answer 1

I have been literally experiencing the same issue for the past 6-7 hours, and finally got this to work. It seems that the parameter that is causing this issue is the 'tempLocation'. Also it looks like you have an additional set of {} within your input and output location paramaters. Try the following:

BODY = {
    "jobName": "{jobname}".format(jobname=JOBNAME),
    "parameters": {
        "customGcsTempLocation": "gs://{bucket}/dts/temp".format(bucket=BUCKET),
        "inputLocations":"{\"location1\": \"projectname:DATASET.table\",\"location2\": \"projectname:DATASET.table\",\"location3\": \"projectname:DATASET.table\",\"location4\": \"gs://bucket/filename.csv\",\"location5\":\"projectname:DATASET.table\",\"location6\":\"projectname:DATASET.table\",\"location7\":\"projectname:DATASET.table\",\"location8\":\"projectname:DATASET.table\",\"location9\": [\"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename\"]}",
        "outputLocations":"{\"projectname:DATASET.table\"}"
    }
}

Let me know if that works for you!

Answer 2

Cristian was correct but there was one other error I had made, posting in case it helps others in similar situations.

I should have excaped the filenames in location9 differently because they are in a list (rather than just being a value) apparently. The below shows how this was done:

\"location9\":\"[\\\"gs://bucket/folder/folder.tsv\\\",\\\"gs://bucket/folder/filename1122.tsv\\\",\\\"gs://bucket/folder/filename1123.tsv\\\",\\\"gs://bucket/folder/filename1124.tsv\\\",\\\"gs://bucket/folder/filename1125.tsv\\\",\\\"gs://bucket/folder/filename1126.tsv\\\",\\\"gs://bucket/folder/filename1127.tsv\\\",\\\"gs://bucket/folder/filename1128.tsv\\\",\\\"gs://bucket/folder/filename1129.tsv\\\",\\\"gs://bucket/folder/filename1201.tsv\\\",\\\"gs://bucket/folder/filename1202.tsv\\\",\\\"gs://bucket/folder/filename1203.tsv\\\",\\\"gs://bucket/folder/filename1204.tsv\\\",\\\"gs://bucket/folder/filename1205.tsv\\\",\\\"gs://bucket/folder/filename1206.tsv\\\",\\\"gs://bucket/folder/filename1207.tsv\\\",\\\"gs://bucket/folder/filename1208.tsv\\\",\\\"gs://bucket/folder/filename1209.tsv\\\",\\\"gs://bucket/folder/filename1210.tsv\\\",\\\"gs://bucket/folder/filename1211.tsv\\\",\\\"gs://bucket/folder/filename1212.tsv\\\"]\"