Airbnb Airflow using all system resources

Question

We\'ve set up Airbnb/Apache Airflow for our ETL using LocalExecutor, and as we\'ve started building more complex DAGs, we\'ve noticed that Airflow has starting usin

Answer 1

the key point is HOW to processing dag files. reduce cpu usage from 80%+ to 30% for scheduler on a 8-core server, i have updated 2 config key,

min_file_process_interval from 0 to 60.
max_threads from 1000 to 50.