apache-airflow-xcom | 易学教程

Airflow: creating & passing list of tables via XCOM (without storing as a file on a drive) and setting up correct dependencies?

阅读更多关于 Airflow: creating & passing list of tables via XCOM (without storing as a file on a drive) and setting up correct dependencies?

问题 Here's the expected flow and dependency setting that I want to achieve: START===>Create table list(only once when DAG is triggered)===> Read & pass table names (via XCOM) ===> Create individual tasks dynamically for each table in the list===> Print table name ===>END Here's the sample code flow: start = DummyOperator( task_id = 'start', dag=dag ) end = DummyOperator( task_id = 'end', dag=dag ) #create table list: def create_source_table_list(dsn, uid, pwd, exclude_table_list, **kwargs): try:

Airflow: Get previous task id in the next task

阅读更多关于 Airflow: Get previous task id in the next task

来源： https://stackoverflow.com/questions/56309773/airflow-get-previous-task-id-in-the-next-task

Airflow: Get previous task id in the next task

阅读更多关于 Airflow: Get previous task id in the next task

来源： https://stackoverflow.com/questions/56309773/airflow-get-previous-task-id-in-the-next-task

Airflow k8s operator xcom - Handshake status 403 Forbidden

阅读更多关于 Airflow k8s operator xcom - Handshake status 403 Forbidden

问题 When I run a docker image using KubernetesPodOperator in Airflow version 1.10 Once the pod finishes the task successfullly, airflow tries to get the xcom value by making a connection to the pod via k8s stream client. Following is the error which I encountered: [2018-12-18 05:29:02,209] {{models.py:1760}} ERROR - (0) Reason: Handshake status 403 Forbidden Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 249, in websocket_call

No such file or directory /airflow/xcom/return.json

阅读更多关于 No such file or directory /airflow/xcom/return.json

问题 created an image contain /airflow/xcom/return.json with chmod +x on all sub-dir Since the log show it cannot find file or directory(tried chmod +x) strtpodbefore = KubernetesPodOperator(namespace='rguonew', image="mydockerimage", name="fail", image_pull_policy="Always", task_id="failing-task", get_logs= True, xcom_push=True, dag=dag )' Here is the log [2019-03-18 20:32:07,007] {logging_mixin.py:95} INFO - [2019-03-18 20:32:07,007] {pod_launcher.py:166} INFO - Running command... cat /airflow

How to pass dynamic arguments Airflow operator?

阅读更多关于 How to pass dynamic arguments Airflow operator?

问题 I am using Airflow to run Spark jobs on Google Cloud Composer. I need to Create cluster (YAML parameters supplied by user) list of spark jobs (job params also supplied by per job YAML) With the Airflow API - I can read YAML files, and push variables across tasks using xcom. But, consider the DataprocClusterCreateOperator() cluster_name project_id zone and a few other arguments are marked as templated. What if I want to pass in other arguments as templated (which are currently not so)? - like

Airflow: How to get the return output of one task to set the dependencies of the downstream tasks to run?

阅读更多关于 Airflow: How to get the return output of one task to set the dependencies of the downstream tasks to run?

问题 We have a kubernetes pod operator that will spit out a python dictionary that will define which further downstream kubernetes pod operators to run along with their dependencies and the environment variables to pass into each operator. How do I get this python dictionary object back into the executor's context (or is it worker's context?) so that airflow can spawn the downstream kubernetes operators? I've looked at BranchOperator and TriggerDagRunOperator and XCOM push/pull and Variable.get

How to pass dynamic arguments Airflow operator?

阅读更多关于 How to pass dynamic arguments Airflow operator?

I am using Airflow to run Spark jobs on Google Cloud Composer. I need to Create cluster (YAML parameters supplied by user) list of spark jobs (job params also supplied by per job YAML) With the Airflow API - I can read YAML files, and push variables across tasks using xcom. But, consider the DataprocClusterCreateOperator() cluster_name project_id zone and a few other arguments are marked as templated. What if I want to pass in other arguments as templated (which are currently not so)? - like image_version , num_workers , worker_machine_type etc? Is there any workaround for this? Not sure what

Airflow k8s operator xcom - Handshake status 403 Forbidden

阅读更多关于 Airflow k8s operator xcom - Handshake status 403 Forbidden

When I run a docker image using KubernetesPodOperator in Airflow version 1.10 Once the pod finishes the task successfullly, airflow tries to get the xcom value by making a connection to the pod via k8s stream client. Following is the error which I encountered: [2018-12-18 05:29:02,209] {{models.py:1760}} ERROR - (0) Reason: Handshake status 403 Forbidden Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 249, in websocket_call client = WSClient(configuration, get_websocket_url(url), headers) File "/usr/local/lib/python3.6/site

Failed to extract xcom from airflow pod - Kubernetes Pod Operator

阅读更多关于 Failed to extract xcom from airflow pod - Kubernetes Pod Operator

问题 While running a DAG which runs a jar using a docker image, xcom_push=True is given which creates another container along with the docker image in a single pod. DAG : jar_task = KubernetesPodOperator( namespace='test', image="path to image", image_pull_secrets="secret", image_pull_policy="Always", node_selectors={"d-type":"na-node-group"}, cmds=["sh","-c",..~running jar here~..], secrets=[secret_file], env_vars=environment_vars, labels={"k8s-app": "airflow"}, name="airflow-pod", config_file