I\'m running servers in a AWS auto scale group. The running servers are behind a load balancer. I\'m using the ELB to mange the auto scaling groups healthchecks. When servers ar
There are two types of Health Check available for Auto Scaling groups:
Given that your system is using an ELB health check, Auto Scaling will trust the results of the ELB health check when determining the health of each EC2 instance. This can be slightly dangerous because, if the instance takes a while to start, the health check could incorrectly mark the instance as Unhealthy. This, in turn, would cause Auto Scaling to terminate the instance and launch a replacement.
To avoid this situation, there is a Health Check Grace Period setting (in seconds) in the Auto Scaling group configuration. This indicates how long Auto Scaling should wait until it starts using the ELB health check (which, in turn, has settings for how often to check and how many checks are required to mark an instance as Healthy/Unhealthy).
So, if your application takes 3 minutes to start, set the Health Check Grace Period to a minimum of 180 seconds (3 minutes). The documentation does not state whether the timing starts from the moment that an instance is marked as "Running" or whether it is when the Status Checks complete, so perform some timing tests to avoid any "bounce" situations.
In fact, I would recommend setting the Health Check Grace Period to a significantly higher value (eg double the amount of time required). This will not impact the operation of your system since a Healthy Instance will start serving traffic as soon as the ELB Health Check is satisfied, which sooner than the Auto Scaling grace period. The worst case is that a genuinely unhealthy instance will be terminated a few minutes later, but this should be a rare occurrence.