Missing log lines when writing to cloudwatch from ECS Docker containers

Question

(Docker container on AWS-ECS exits before all the logs are printed to CloudWatch Logs) Why are some streams of a CloudWatch Logs Group incomplete (i.e., the Fargate Docker Conta

Answer 1

I had the same problem with flushing logs to CloudWatch.

Following asavoy's answer I switched from exec form to shell form of the ENTRYPOINT and added a 10 sec sleep at the end.

Before:

ENTRYPOINT ["java","-jar","/app.jar"]

After:

ENTRYPOINT java -jar /app.jar; sleep 10

Answer 2

UPDATE: This now appears to be fixed, so there is no need to implement the workaround described below

---

I've seen the same behaviour when using ECS Fargate containers to run Python scripts - and had the same resulting frustration!

I think it's due to CloudWatch Logs Agent publishing log events in batches:

How are log events batched?

A batch becomes full and is published when any of the following conditions are met:

1. The buffer_duration amount of time has passed since the first log event was added.

2. Less than batch_size of log events have been accumulated but adding the new log event exceeds the batch_size.

3. The number of log events has reached batch_count.

4. Log events from the batch don't span more than 24 hours, but adding the new log event exceeds the 24 hours constraint.

(Reference: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html)

So a possible explanation is that log events are buffered by the agent but not yet published when the ECS task is stopped. (And if so, that seems like an ECS issue - any AWS ECS engineers willing to give their perspective on this...?)

There doesn't seem to be a direct way to ensure the logs are published, but it does suggest one could wait at least buffer_duration seconds (by default, 5 seconds), and any prior logs should be published.

With a bit of testing that I'll describe below, here's a workaround I landed on. A shell script run_then_wait.sh wraps the command to trigger the Python script, to add a sleep after the script completes.

Dockerfile

FROM python:3.7-alpine
ADD run_then_wait.sh .
ADD main.py .

# The original command
# ENTRYPOINT ["python", "main.py"]

# To run the original command and then wait
ENTRYPOINT ["sh", "run_then_wait.sh", "python", "main.py"]

run_then_wait.sh

#!/bin/sh
set -e

# Wait 10 seconds on exit: twice the `buffer_duration` default of 5 seconds
trap 'echo "Waiting for logs to flush to CloudWatch Logs..."; sleep 10' EXIT

# Run the given command
"$@"

main.py

import logging
import time

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

if __name__ == "__main__":
    # After testing some random values, had most luck to induce the
    # issue by sleeping 9 seconds here; would occur ~30% of the time
    time.sleep(9)
    logger.info("Hello world")

Hopefully the approach can be adapted to your situation. You could also implement the sleep inside your script, but it can be trickier to ensure it happens regardless of how it terminates.

It's hard to prove that the proposed explanation is accurate, so I used the above code to test whether the workaround was effective. The test was the original command vs. with run_then_wait.sh, 30 runs each. The results were that the issue was observed 30% of the time, vs 0% of the time, respectively. Hope this is similarly effective for you!

Answer 3

I observed this as well. It must be an ECS bug?

My workaround (Python 3.7):

import atexit
from time import sleep

atexit.register(finalizer)

def finalizer():
    logger.info("All tasks have finished. Exiting.")
    # Workaround:
    # Fargate will exit and final batch of CloudWatch logs will be lost
    sleep(10)

Answer 4

Just contacted AWS support about this issue and here is their response:

...

Based on that case, I can see that this occurs for containers in a Fargate Task that exit quickly after outputting to stdout/stderr. It seems to be related to how the awslogs driver works, and how Docker in Fargate communicates to the CW endpoint.

Looking at our internal tickets for the same, I can see that our service team are still working to get a permanent resolution for this reported bug. Unfortunately, there is no ETA shared for when the fix will be deployed. However, I've taken this opportunity to add this case to the internal ticket to inform the team of the similar and try to expedite the process

In the meantime, this can be avoided by extending the lifetime of the exiting container by adding a delay (~>10 seconds) between the logging output of the application and the exit of the process (exit of the container).

...

Update: Contacted AWS around August 1st, 2019, they say this issue has been fixed.