Cloudwatch failedinvocation error no logs available

Question

I have set up a Cloudwatch rule event where an ECS task definition is started when a previous task definition is completed.

I can see the event triggers the task de

Answer 1

If the rule has been successfully triggered, but the invocation on the target failed, you should see a trace of the API call in the Event History inside the AWS CloudTrail looking at the errorCode and errorMessage properties:

{
   [..]
   "errorCode": "InvalidInputException",
   "errorMessage": "Artifacts type is required",
   [..]
}

Answer 2

In case other people come here looking for the setup necessary to make this work for a task in Fargate. There is some extra configuration in addition to Stefano's answer. Running tasks in Fargate requires setting up an execution role, so you need to enable the CloudWatchEventECSRole to use it. Add this statement to that role:

{
    "Effect": "Allow",
    "Action": "iam:PassRole",
    "Resource": [
        "arn:aws:iam::<account>:role/<executionRole>"
    ]
}

Answer 3

I spent ages trying to troubleshoot this, when creating an ECS scheduled task via the command line the task was created but never started. Thanks for this post, I discovered by looking at the EventHistory in CloudTrail that the ECS instances had all died and there were no EC2 instances running!

{
   [..]
 "errorCode": "InvalidParameterException",
 "errorMessage": "No Container Instances were found in your cluster.",
   [..]
}

Answer 4

CloudTrail logs helped. event Name is RunTask. The issue was: "errorCode": "InvalidParameterException", "errorMessage": "Override for container named rds-task is not a container in the TaskDefinition.",

The AWS documentation for debugging CloudWatch events is here:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CWE_Troubleshooting.html

I opened a PR to add documentation for debugging failed ECS Task Invocations from CloudWatch Events:
https://github.com/awsdocs/amazon-cloudwatch-events-user-guide/pull/12/files

Answer 5

This problem was due to not setting the principle services to include events.amazonaws.com. The task couldn't assume the role.

Shame aws doesn't have better logging for failedinvocations.

Answer 6

I too was not seeing my lambda executing, but I did find evidence of FailedInvocations in CloudWatch Events (but only via the Event Rule Metrics link, which took me to https://console.aws.amazon.com/cloudwatch/home?region={your_aws_region}#metricsV2:graph=~();query=~'*7bAWS*2fEvents*2cRuleName*7d*2{Lambda_Physical_ID})

I was not seeing the "trigger" in the console either so I took a step back, decided to do a more "simple" SAM deploy with the Events property set, then looked at the processed template to determine how it was done in that case. Below is what I ended up using to implement "EventBridge" to have a ScheduledEvent fire my Lambda (alias in my case, which is why I discovered this).

Simple SAM approach to scheduled invocations

(Add this property to your AWS::Serverless::Function)

Events:
  InvokeMyLambda:
    Type: Schedule
    Properties:
      Schedule: rate(1 minute)
      Description: Run SampleLambdaFunction once every minute.
      Enabled: True

By looking at the converted template in CloudFormation and comparing to the version without Events, I was able to identify not on the expected AWS::Events::Rule (which is what I expected to see invocing the lambad), but I also saw AWS::Lambda::Permission.

Hopefully this is what you all are needing as well to get invocations working (and not needing the missing logs to see why) :P

Working approach

MyLambdaScheduledEvent:
  Type: AWS::Events::Rule
  Properties:
    Name: MyLambdaScheduledEvent
    EventBusName: "default"
    State: ENABLED
    ScheduleExpression: rate(5 minutes) # same as cron(0/5 * * * ? *)
    Description: Run MyLambda once every 5 minutes.
    Targets:
    - Id: EventMyLambdaScheduled
      Arn: !Ref MyLambda
MyLambdaScheduledEventPermission:
  Type: AWS::Lambda::Permission
  Properties:
    Action: lambda:InvokeFunction
    Principal: events.amazonaws.com
    FunctionName: !Ref MyLambda
    SourceArn: !GetAtt MyLambdaScheduledEvent.Arn