ECS Fargate task not applying role

Question

I have an ECS Fargate task running that has a role attached to it. This role has the S3FullAccess policy (and AssumeRole trusted partnership with ECS service).

Howev

Answer 1

Amazon ECS container credentials– loaded from the Amazon ECS if the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set.

You define the IAM role to use in your task definitions, or you can use a taskRoleArn override when running a task manually with the RunTask API operation. The Amazon ECS agent receives a payload message for starting the task with additional fields that contain the role credentials. The Amazon ECS agent sets a unique task credential ID as an identification token and updates its internal credential cache so that the identification token for the task points to the role credentials that are received in the payload. The Amazon ECS agent populates the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable in the Env object (available with the docker inspect container_id command) for all containers that belong to this task with the following relative URI: /credential_provider_version/credentials?id=task_credential_id.

Terraform code:

resource "aws_iam_role" "AmazonS3ServiceForECSTask" {
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": [
          "ecs-tasks.amazonaws.com"
        ]
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

data "aws_iam_policy_document" "bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.AmazonS3ServiceForECSTask.arn]
    }

    actions = [
      "s3:ListBucket",
      "s3:GetBucketLocation",
    ]

    resources = [
      "arn:aws:s3:::${var.bucket_name}",
    ]
  }
  statement {
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.AmazonS3ServiceForECSTask.arn]
    }

    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:PutObjectAcl",
      "s3:ListMultipartUploadParts",
      "s3:AbortMultipartUpload",
    ]

    resources = [
      "arn:aws:s3:::${var.bucket_name}/*",
    ]
  }
}

resource "aws_ecs_task_definition" "sephora_ba_ecs_task_definition" {
  task_role_arn               = aws_iam_role.AmazonS3ServiceForECSTask.arn
  execution_role_arn          = aws_iam_role.ECS-TaskExecution.arn
  family                      = "${var.family}"
  network_mode                = var.network_mode[var.launch_type]
  requires_compatibilities    = var.requires_compatibilities
  cpu                         = var.task_cpu[terraform.workspace]
  memory                      = var.task_memory[terraform.workspace]
  container_definitions       = module.ecs-container-definition.json
}

Answer 2

I've just had a similar issue and I think it's probably due to your program being unable to access the role's credentials that are exposed by the Instance Metadata service.

Specifically, there's an environment variable called AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and its value is what's needed by the AWS SDKs to use the task role. The ECS Container Agent sets it when your task starts, and it is exposed to the container's main process that has process ID 1. If your program isn't running as such, it might not being seeing the env var and so explaining the access denied error.

Depending on how your program is running, there'll be different ways to share the env var.

I had the issue inside ssh login shells (BTW you can ssh into Fargate tasks by running sshd) so in my Docker entrypoint script I inserted somewhere:

# To share the env var with login shells
echo "export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" >> /root/.profile

In other cases it might work to add to your Docker entrypoint script:

# To export the env var for use by child processes
export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

References:

• IAM Roles for Tasks - docs explaining the env var relating to the role
• AWS Forum post - where someone explains these workarounds in greater detail