airflow

问题 I have created a Airflow plugin which creates a new Menu named Test Plugin and add a submenu Test View so clicking the Test View open the page successfully and show me the content in test.html page. Currently Airflow displays the landing page on the url http://localhost:8080/admin/ with all Dags listed. My requirement is to show this test.html page as the landing page/home page. The structure of is as follows: -AIRFLOW_HOME/plugins/templates/test_plugin/test.html -AIRFLOW_HOME/plugins/test

问题 I have created a Airflow plugin which creates a new Menu named Test Plugin and add a submenu Test View so clicking the Test View open the page successfully and show me the content in test.html page. Currently Airflow displays the landing page on the url http://localhost:8080/admin/ with all Dags listed. My requirement is to show this test.html page as the landing page/home page. The structure of is as follows: -AIRFLOW_HOME/plugins/templates/test_plugin/test.html -AIRFLOW_HOME/plugins/test

问题 I have a single container Airflow setup running on Marathon, using the LocalExecutor. I have a health check running that pings the /health endpoint on the Airflow webserver. It currently has 5 cpus allocated to it and the webserver is running 4 Gunicorn. Last night I had about 25 tasks running concurrently. This caused the health check to fail w/o a helpful error message. The container just received a SIGTERM. I was wondering if anyone could suggest a likely culprit for what caused the health

问题 I'm using airflow(Google composer), but experienced some exceptions below TypeError: can't pickle _thread.RLock objects Ooops. ____/ ( ( ) ) \___ /( ( ( ) _ )) ) )\ (( ( )( ) ) ( ) ) ((/ ( _( ) ( _) ) ( () ) ) ( ( ( (_) (( ( ) .((_ ) . )_ ( ( ) ( ( ) ) ) . ) ( ) ( ( ( ( ) ( _ ( _) ). ) . ) ) ( ) ( ( ( ) ( ) ( )) ) _)( ) ) ) ( ( ( \ ) ( (_ ( ) ( ) ) ) ) )) ( ) ( ( ( ( (_ ( ) ( _ ) ) ( ) ) ) ( ( ( ( ( ) (_ ) ) ) _) ) _( ( ) (( ( )( ( _ ) _) _(_ ( (_ ) (_((__(_(__(( ( ( | ) ) ) )_))__))_)___) ((

问题 I need to restart Airflow. I want to make sure I do it when it's idle, so I that I don't interrupt a job by restarting the worker component of Airflow. How do I see what DAGs are running? I don't see anything in the UI that would list currently running DAGs. I don't see any command in the airflow CLI to list currently running DAGs. I found airflow shell that lets me connect to the DB, but I don't know enough about Airflow internals to know where to look to see what's running. 回答1: You can

问题 I want to create task which will be update columns rows and send mail for every line in data table. At the moment I create task which download the data from main table. I cannot create tasks for every line in temp data table. Could you tell what I doing wrong and how I can generate and run tasks in lopp? from datetime import datetime, timedelta import airflow from airflow import DAG from airflow.contrib.operators.bigquery_operator import BigQueryOperator from airflow.contrib.operators

问题 I want to create task which will be update columns rows and send mail for every line in data table. At the moment I create task which download the data from main table. I cannot create tasks for every line in temp data table. Could you tell what I doing wrong and how I can generate and run tasks in lopp? from datetime import datetime, timedelta import airflow from airflow import DAG from airflow.contrib.operators.bigquery_operator import BigQueryOperator from airflow.contrib.operators

问题 For reasons acceptable to me, I am trying to dynamically generate ExternalTaskSensor tasks with different execution_date_fn in each iteration. Callable provided to execution_date_fn kwarg requires to have dt as input and provide execution_date as output, which I am writing down as a lambda function, e.g. lambda dt: get_execution_date(i) . I noticed that execution_date_fn provided as a lambda function in a loop results in unexpected behaviour - all generated tasks have the same execution_date

问题 For reasons acceptable to me, I am trying to dynamically generate ExternalTaskSensor tasks with different execution_date_fn in each iteration. Callable provided to execution_date_fn kwarg requires to have dt as input and provide execution_date as output, which I am writing down as a lambda function, e.g. lambda dt: get_execution_date(i) . I noticed that execution_date_fn provided as a lambda function in a loop results in unexpected behaviour - all generated tasks have the same execution_date

问题 We are using Airflow(1.10.3) with celery executor(4.1.1 (latentcall)) and broker SQS. While debugging an issue we tried our hands on celery CLI and found out that SQS broker is not supported for any of the inspect commands or monitoring tool eg. Flower. Is there any way we can monitor the tasks or events on celery workers? We have tried curses monitor as below: celery events -b sqs:// But it shows no worker discovered,no tasks selected. Inspect Commands directly shows: Availability: RabbitMQ