问题
Fellow coders,
For a project I'm interested in using Google Cloud Composer to handle several workflows that consist of operations that can be shared between workflows.
It seems to me that Cloud Functions are a perfect way of performing tasks as these operations in a Composer DAG.
For what I understood of it, I would need an operator that invokes a cloud function with data that is specific for the task in the specific DAG.
I found a Google Cloud Function operator in the Airflow documentation, however, these are only for deploying and deleting cloud functions, but not for invoking them.
A lot has been written about invoking DAGs from a cloud function, but nothing seems to be written about using cloud functions as operations within a DAG.
Example use case:
Every time a document is placed in a certain bucket I want to start a DAG workflow to analyse this document.
This DAG can consist of various tasks, such as extraction of the sender of the document, classification of a logo, or searching for specific words. For these separate tasks, I want to create separate cloud functions that are stitched together in a DAG to compose my workflows.
Question:
How to invoke cloud functions from within a Google Composer DAG?
Do people have experience with this or have sample code available?
Thanks in advance.
回答1:
HTTP Triggers can be used to run Cloud Functions, so you can invoke them from a DAG using the HTTP operator. The DAG runs the task, that calls the Cloud Function trigger, than runs the function.
来源：https://stackoverflow.com/questions/57485742/using-cloud-functions-as-operators-in-a-gc-composer-dag