Amazon EC2 autoscaling down with graceful shutdown?

Question

We\'re looking at using EC2 autoscaling to deal with spikes in load. In our case we want to scale up instances based on an SQS queue size and then down scale with the queue

Answer 1

On most newer AMI's, the machines are given the equivalent to a 'halt' (or 'shutdown -h now' command so that the services are gracefully shut down. As long as your program plays nicely with the startup/shutdown scripts, you should be fine -- but, if your program takes more than 20 seconds to terminate, you may experience that amazon will kill the instance completely.

Amazon's documentation with regards to their autoscaling doesn't specify the termination process, but, AWS's documentation for ec2 in general does contain about what happens during the termination process -- that the machines is given a 'shutdown' command, and the default shutdown time on most systems is 30 seconds.

Answer 2

In mid 2014 AWS introduced 'lifecycle hooks' which allows for full control of the termination process.

Our high level down scale process is:

• Auto Scaling sends a message to a SQS queue with an instance ID
• Controller app picks up the message
• Controller app issues a 'stop instance' request
• Controller app re-queues the SQS message while the instance is stopping
• Controller app picks up the message again, checks if the instance has stopped (or re-queues the message to try again later)
• Controller app notifies Auto Scaling to 'PROCEED' with the termination
• Controller app deletes the message from the SQS queue

More details: http://docs.aws.amazon.com/autoscaling/latest/userguide/lifecycle-hooks.html

Answer 3

use replaceunhealty option in autoscaling.

refer: http://alestic.com/2011/11/ec2-schedule-instance

particularly see this comment.