Airbnb Airflow using all system resources

Question

We\'ve set up Airbnb/Apache Airflow for our ETL using LocalExecutor, and as we\'ve started building more complex DAGs, we\'ve noticed that Airflow has starting usin

Answer 1

I just ran into an issue like this. Airflow was consuming roughly a full vCPU in a t2.xlarge instance, with the vast majority of this coming from the scheduler container. Checking the scheduler logs, I could see that it was processing my single DAG more than once a second even though it only runs once a day.

I found that the MIN_FILE_PROCESS_INTERVAL was set to the default value of 0, so the scheduler was looping over the DAG. I changed the process interval to 65 seconds, and Airflow now uses less than 10 percent of a vCPU in a t2.medium instance.