Airflow 1.9.0 is queuing but not launching tasks

Question

Airflow is randomly not running queued tasks some tasks dont even get queued status. I keep seeing below in the scheduler logs

 [2018-02-28 02:24:58,780] {jobs.py:1077} INFO - No tasks to consider for execution.

I do see tasks in database that either have no status or queued status but they never get started.

The airflow setup is running https://github.com/puckel/docker-airflow on ECS with Redis. There are 4 scheduler threads and 4 Celery worker tasks. For the tasks that are not running are showing in queued state (grey icon) when hovering over the task icon operator is null and task details says:

    All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless:- The scheduler is down or under heavy load

Metrics on scheduler do not show heavy load. The dag is very simple with 2 independent tasks only dependent on last run. There are also tasks in the same dag that are stuck with no status (white icon).

Interesting thing to notice is when I restart the scheduler tasks change to running state.

Answer 1

Airflow can be a bit tricky to setup.

• Do you have the airflow scheduler running?
• Do you have the airflow webserver running?
• Have you checked that all DAGs you want to run are set to On in the web ui?
• Do all the DAGs you want to run have a start date which is in the past?
• Do all the DAGs you want to run have a proper schedule which is shown in the web ui?
• If nothing else works, you can use the web ui to click on the dag, then on Graph View. Now select the first task and click on Task Instance. In the paragraph Task Instance Details you will see why a DAG is waiting or not running.

I've had for instance a DAG which was wrongly set to depends_on_past: True which forbid the current instance to start correctly.

Also a great resource directly in the docs, which has a few more hints: Why isn't my task getting scheduled?.

Answer 2

I'm running a fork of the puckel/docker-airflow repo as well, mostly on Airflow 1.8 for about a year with 10M+ task instances. I think the issue persists in 1.9, but I'm not positive.

For whatever reason, there seems to be a long-standing issue with the Airflow scheduler where performance degrades over time. I've reviewed the scheduler code, but I'm still unclear on what exactly happens differently on a fresh start to kick it back into scheduling normally. One major difference is that scheduled and queued task states are rebuilt.

Scheduler Basics in the Airflow wiki provides a concise reference on how the scheduler works and its various states.

Most people solve the scheduler diminishing throughput problem by restarting the scheduler regularly. I've found success at a 1-hour interval personally, but have seen as frequently as every 5-10 minutes used too. Your task volume, task duration, and parallelism settings are worth considering when experimenting with a restart interval.

For more info see:

• Airflow: Tips, Tricks, and Pitfalls (section "The scheduler should be restarted frequently")
• Bug 1286825 - Airflow scheduler stopped working silently
• Airflow at WePay (section "Restart everything when deploying DAG changes.")

This used to be addressed by restarting every X runs using the SCHEDULER_RUNS config setting, although that setting was recently removed from the default systemd scripts.

You might also consider posting to the Airflow dev mailing list. I know this has been discussed there a few times and one of the core contributors may be able to provide additional context.

Related Questions

• Airflow tasks get stuck at "queued" status and never gets running (especially see Bolke's answer here)
• Jobs not executing via Airflow that runs celery with RabbitMQ

Answer 3

I am facing the issue today and found that bullet point 4 from tobi6 answer below worked out and resolved the issue

*'Do all the DAGs you want to run have a start date which is in the past?'*

I am using airflow version v1.10.3

Answer 4

My problem was one step further, in addition to my tasks being queued, I couldn't see any of my celery workers on the Flower UI. The solution was that, since I was running my celery worker as root I had to make changes in my ~/.bashrc file.

The following steps made it work:

1. Add export C_FORCE_ROOT=true to your ~/.bashrc file
2. source ~/.bashrc
3. Run worker : nohup airflow worker $* >> ~/airflow/logs/worker.logs &

Check your Flower UI at http://{HOST}:5555

Answer 5

One more thing to check is whether "the concurrency parameter of your DAG reached?".

I'd experienced the same situation when some task was shown as NO STATUS.

It turned out that my File_Sensor tasks were run with timeout set up to 1 week, while DAG time out was only 5 hours. That leaded to the case when the Files were missing, many sensors tasked were running at the same time. Which results the concurrency overloaded!

The depending tasks couldn't be started before the sensor task succeed, when the dag timeout, they got NO STATUS.

My solution:

• Carefully set tasks and DAG timeout
• Increase dag_concurrency in airflow.cfg file in AIRFLOW_HOME folder.

Please refer to the docs. https://airflow.apache.org/faq.html#why-isn-t-my-task-getting-scheduled

Answer 6

I also had a similar issue, but it is mostly related to SubDagOperator with more than 3000 task instances in total (30 tasks * 44 subdag tasks).

What I found out is that airflow scheduler mainly responsible for putting your scheduled tasks in to "Queued Slots" (pool), while airflow celery workers is the one who pick up your queued task and put it into the "Used Slots" (pool) and run it.

Based on your description, your scheduler should work fine. I suggest you check your "celery workers" log to see whether there is any error, or restart it to see whether it helps or not. I experienced some issues that celery workers normally go on strike for a few minutes then start working again (especially on SubDagOperator)