How do you keep your airflow scheduler running in AWS EC2 while exiting ssh?

问题

Hi I'm using Airflow and put my airflow project in EC2. However, how would one keep the airflow scheduler running while my mac goes sleep or exiting ssh?

回答1:

You have a few options, but none will keep it active on a sleeping laptop. On a server: Can use --daemon to run as daemon: airflow scheduler --daemon Or, maybe run in background: airflow scheduler >& log.txt & Or, run inside 'screen' as above, then detach from screen using ctrl-a d, reattach as needed using 'screen -r'. That would work on an ssh connection.

回答2:

I use nohup to keep the scheduler running and redirect the output to a log file like so:

nohup airflow scheduler >> ${AIRFLOW_HOME}/logs/scheduler.log 2>&1 &

Note: Assuming you are running the scheduler here on your EC2 instance and not on your laptop.

回答3:

The most robust solution would be to register it as a service on your EC2 instance. Airflow provides systemd and upstart scripts for that (https://github.com/apache/incubator-airflow/tree/master/scripts/systemd and https://github.com/apache/incubator-airflow/tree/master/scripts/upstart). For Amazon Linux, you'd need the upstart scripts, and for e.g. Ubuntu, you would use the systemd scripts.

Registering it as a system service is much more robust because Airflow will be started upon reboot or when it crashes. This is not the case when you use e.g. nohup like other people suggest here.

回答4:

In case you need more details on running it as deamon i.e. detach completely from terminal and redirecting stdout and stderr, here is an example:

airflow webserver -p 8080 -D --pid /your-path/airflow-webserver.pid --stdout /your-path/airflow-webserver.out --stderr /your-path/airflow-webserver.err  

airflow scheduler -D --pid /your-path/airflow-scheduler.pid —stdout /your-path/airflow-scheduler.out --stderr /your-path/airflow-scheduler.err 

来源：https://stackoverflow.com/questions/45168300/how-do-you-keep-your-airflow-scheduler-running-in-aws-ec2-while-exiting-ssh