Airflow latency between tasks

Question

As you can see in the image : airflow is making too much time between tasks execution ? it

Answer 1

It is by design. For instance I use Airflow to perform large workflows where some tasks can take a really long time. Airflow is not meant for tasks that will take seconds to execute, it can be used for that of course but might not be the most suitable tool.

With that said there is not much that you can do since you already found out the key settings to configure.

Additionally you might want to try to increase the number of threads of the scheduler:

   [scheduler]
   max_threads = 4

This can alternatively be done by setting the environment variable:

AIRFLOW__SCHEDULER__MAX_THREADS=4

However do not count on the latency to decrease that much.