Using PYTHON to run a Google Dataflow Template

Question

I want to execute a Google Dataflow Template using PYTHON. Actually, I have been executing Dataflow Templates using the Dataflow REST API or the Cloud Fun

Answer 1

You can do that using the template launch method from the Dataflow API Client Library for Python like so:

import googleapiclient.discovery
from oauth2client.client import GoogleCredentials

project = PROJECT_ID
location = LOCATION

credentials = GoogleCredentials.get_application_default()

dataflow = googleapiclient.discovery.build('dataflow', 'v1b3', credentials=credentials)
result = dataflow.projects().templates().launch(
        projectId=project,
        body={
          "environment": {
            "zone": "us-central1-f",
            "tempLocation": "gs://{{my-cloud-storage-bucket}}/tmp"
          },
          "parameters": {
              "inputLocations" : "{\"location1\":\"gs://{{my-cloud-storage-bucket}}/my-folder/**/*\"}",
              "outputLocations": "{\"location1\":\"gs://{{my-cloud-storage-bucket}}/my-output/output.csv\"}"
          },
          "jobName": SOME_NAME
        },
        gcsPath = PATH_TO_TEMPLATE
).execute()