Airflow latency between tasks

Question

As you can see in the image : airflow is making too much time between tasks execution ? it

Answer 1

Thirty seconds is fairly high for inter-task latency. In well-tuned environments I've seen, ~4-6 seconds between a task and a dependent task has been a fairly reasonable lower bound, even for environments with many thousands of DAGs.

As you've already stated, increasing the scheduler heartbeat (scheduler_heartbeat_sec) and the number of threads the scheduler has (scheduler.max_threads) are the best to decrease scheduling delays. If your tasks are blocked on other conditions (which you can check in logs; core.logging_level = DEBUG for even more information), then you should resolve those first.

If you've adjusted both the scheduler heartbeat and the number of worker threads and you still see high scheduling delays, then you may need to consider using a more powerful machine.