Running `airflow scheduler` launches 33 scheduler processes

Question

When using LocalExecutor with a MySQL backend, running airflow scheduler on my Centos 6 box creates 33 scheduler processes, e.g.

Answer 1

Yes this is normal. These are scheduler processes. You can control this using below parameter in airflow.cfg

# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32

These are spawned from scheduler whose pid can be found in airflow-scheduler.pid file

so 32+1=33 processes that you are seeing.

Hope this clears out your doubt.

Cheers!

Answer 2

As of v1.10.3, this is what I found. My settings are:

parallelism = 32
max_threads = 4

There are a total of

• 1 (main) +
• 32 (executors) +
• 1 (dag_processor_manager) +
• 4 (dag processors)

= 38 processes!