Airbnb Airflow using all system resources

Question

We\'ve set up Airbnb/Apache Airflow for our ETL using LocalExecutor, and as we\'ve started building more complex DAGs, we\'ve noticed that Airflow has starting usin

Answer 1

I tried to run Airflow on a AWS t2.micro instance (1vcpu, 1gb of memory, eligible for free tier), and had the same issue : the worker consumed 100% of the cpu and consumed all available memory.

The EC2 instance was totally stuck and unusable, of course Airflow didn't working.

So I created a 4GB swap file using the method described here. With the swap, no more issues, Airflow was fully functionnal. Of course, with only one vcpu, you cannot expect incredible performances, but it runs.