I want to execute a Google Dataflow Template using PYTHON. Actually, I have been executing Dataflow Templates using the Dataflow REST API
or the Cloud Fun
-
You can do that using the template launch method from the Dataflow API Client Library for Python like so:
import googleapiclient.discovery
from oauth2client.client import GoogleCredentials
project = PROJECT_ID
location = LOCATION
credentials = GoogleCredentials.get_application_default()
dataflow = googleapiclient.discovery.build('dataflow', 'v1b3', credentials=credentials)
result = dataflow.projects().templates().launch(
projectId=project,
body={
"environment": {
"zone": "us-central1-f",
"tempLocation": "gs://{{my-cloud-storage-bucket}}/tmp"
},
"parameters": {
"inputLocations" : "{\"location1\":\"gs://{{my-cloud-storage-bucket}}/my-folder/**/*\"}",
"outputLocations": "{\"location1\":\"gs://{{my-cloud-storage-bucket}}/my-output/output.csv\"}"
},
"jobName": SOME_NAME
},
gcsPath = PATH_TO_TEMPLATE
).execute()
讨论(0)
- 热议问题