Avoiding INSUFFICIENT DATA in Cloudwatch?

Question

I have alarms set up to tell me when my load balancers are throwing 5xxs using the HTTPCode_Backend_5XX metric with the sum statistic. The issue is

Answer 1

I asked for this in the AWS forums two years ago :-( https://forums.aws.amazon.com/thread.jspa?threadID=153753&tstart=0

Unfortunately you cannot create notifications based on specific state changes (in your case you want a notification when state changes from ALARM to OK, but not when state changes from INSUFFICIENT to OK). I can only suggest that you also ask for it and hopefully it will eventually be added.

For metrics that are often in the INSUFFICIENT state I generally just create notifications for ALARMS and I don't have notifications on OK for these metrics - if I want to confirm that things are OK I use the AWS mobile app to check on things and see if they have resolved.

Answer 2

We had a similar issue for some of our alarms. You can actually avoid this behaviour with some work, if you really want to deal with the overhead.

What we have done is, instead of sending SNS notifications directly to e-mails, we have created a lambda function and triggered it once we have the notification in the SNS topic.

This way, you will have more control over the actions you can take once the alarms are triggered. As the context will provide you old state value as well.

The good news is, there is already a lambda template to get started. https://aws.amazon.com/blogs/aws/new-slack-integration-blueprints-for-aws-lambda/

Just pick the one that is designed to send cloudwatch alarms to slack. You can then modify the code as you wish, either dismiss the slack part and just use emails, or keep it with slack. (which is what we did and it works like a charm)

Answer 3

As of March 2017, you can treat missing data as acceptable. This will prevent the alarm from being marked as INSUFFICIENT.

You can also set this in CloudFormation using the TreatMissingData property.