问题
I am trying to start my docker containers using AWS ECS. When i run the task, i see the status as Pending and then immediately i see that my task status is in stopped. But when i try to run the same docker container manually by SSH into that box, i am able to start the container. I am not able to get my docker container up using the ECS agent. i also tried SSH into that box and looked at all the ECS logs which is not helpful in tracking down this issue. Here is what i did step by step
- Created my docker containers locally and tested the same.
- Uploaded the working docker container image to the AWS ECR by following the AWS getting started documentation for ECS.
- Created task definition for my docker containers.
- Created a cluster configuration to run my task
- I ran the task and selected the cluster where the task should be running
I see that task status is pending and immediately status changed to stopped.
I am not sure about how to troubleshoot this issue. Also from the AWS UI, i see that docker container exited with Exit 0.
Here is my task definition which i have created
Task Definition JSON
{
"requiresAttributes": [
{
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth",
"targetId": null,
"targetType": null
},
{
"value": null,
"name": "com.amazonaws.ecs.capability.task-iam-role",
"targetId": null,
"targetType": null
}
],
"taskDefinitionArn": "arn:aws:ecs:us-east-2:####################:taskdefinition/app1:3",
"networkMode": "bridge",
"status": "ACTIVE",
"revision": 3,
"taskRoleArn": "arn:aws:iam::##################################:role/EC2ContainerServiceFullAccess",
"containerDefinitions": [
{
"volumesFrom": [],
"memory": 6000,
"extraHosts": null,
"linuxParameters": null,
"dnsServers": null,
"disableNetworking": null,
"dnsSearchDomains": null,
"portMappings": [
{
"hostPort": 8091,
"containerPort": 8091,
"protocol": "tcp"
}
],
"hostname": null,
"essential": true,
"entryPoint": [
"/docker-entrypoint.sh"
],
"mountPoints": [],
"name": "tomcat-container",
"ulimits": null,
"dockerSecurityOptions": null,
"environment": [
{
"name": "JAVA_OPTS",
"value": "-Xms2G -Xmx2G"
}
],
"links": null,
"workingDirectory": null,
"readonlyRootFilesystem": null,
"image": "##################################/app1:latest",
"command": [
"/bin/bash"
],
"user": null,
"dockerLabels": null,
"logConfiguration": null,
"cpu": 0,
"privileged": null,
"memoryReservation": null
},
{
"volumesFrom": [],
"memory": 512,
"extraHosts": null,
"linuxParameters": null,
"dnsServers": null,
"disableNetworking": null,
"dnsSearchDomains": null,
"portMappings": [
{
"hostPort": 8082,
"containerPort": 8082,
"protocol": "tcp"
}
],
"hostname": null,
"essential": true,
"entryPoint": [
"/docker-entrypoint.sh"
],
"mountPoints": [],
"name": "app2",
"ulimits": null,
"dockerSecurityOptions": null,
"environment": [],
"links": null,
"workingDirectory": null,
"readonlyRootFilesystem": null,
"image": "##################################/app2:latest",
"command": [
"/bin/bash"
],
"user": null,
"dockerLabels": null,
"logConfiguration": null,
"cpu": 0,
"privileged": null,
"memoryReservation": null
}
],
"placementConstraints": [],
"volumes": [
{
"host": {
"sourcePath": "/usr/local/app1/"
},
"name": "hostToContainer"
}
],
"family": "app1"
}
I am kind of stuck with this issue. Does anyone see any other way to fix this issue ?
Note that i am able to bring my application up and running by manually running my docker containers by doing ssh into the EC2 instance. But doing this way, i have a problem of all the docker containers deployed into the EC2 instance is deleted when i stop the EC2 instance. So i think my problem will be solved if i am able to run the docker containers by running the task definition.
I am attaching the logs present in the ecs-agent.logs
2017-11-08T07:01:04Z [INFO] Saving state! module="statemanager"
2017-11-08T07:01:04Z [INFO] Waiting for any previous stops to complete module="TaskEngine" task="csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (NONE->RUNNING),cs_docServer_container (NONE->RUNNING),]" seqnum="19"
2017-11-08T07:01:04Z [INFO] Wait over; ready to move towards status: RUNNING module="TaskEngine" task="csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (NONE->RUNNING),cs_docServer_container (NONE->RUNNING),]"
2017-11-08T07:01:04Z [INFO] Pulling container cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (NONE->RUNNING) concurrently. Task: csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (NONE->RUNNING),cs_docServer_container (NONE->RUNNING),]
2017-11-08T07:01:04Z [INFO] Pulling container cs_tomcat_container(image2YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY) (NONE->RUNNING) concurrently. Task: csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (NONE->RUNNING),cs_docServer_container (NONE->RUNNING),]
2017-11-08T07:01:04Z [INFO] Updating container reference cs_docServer_container in Image State - sha256:cde23d8e79adfa91d9068188bfc184abcc875aac07a0fd80311a2087eb4d196f
2017-11-08T07:01:04Z [INFO] Saving state! module="statemanager"
2017-11-08T07:01:04Z [INFO] Finished pulling container image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx in 153.23238ms. Task: csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (NONE->RUNNING),cs_docServer_container (PULLED->RUNNING),]
2017-11-08T07:01:04Z [INFO] Updating container reference cs_tomcat_container in Image State - sha256:666425a4dd65d237e8a4c5210836e808aa3831d85d5f0e1856b36ed5920b8575
2017-11-08T07:01:04Z [INFO] Saving state! module="statemanager"
2017-11-08T07:01:04Z [INFO] Finished pulling container image2YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY in 522.620398ms. Task: csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (PULLED->RUNNING),cs_docServer_container (PULLED->RUNNING),]
2017-11-08T07:01:04Z [INFO] Creating container module="TaskEngine" task="csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (PULLED->RUNNING),cs_docServer_container (PULLED->RUNNING),]" container="cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (PULLED->RUNNING)"
2017-11-08T07:01:04Z [INFO] Creating container module="TaskEngine" task="csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (PULLED->RUNNING),cs_docServer_container (PULLED->RUNNING),]" container="cs_tomcat_container(image2YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY) (PULLED->RUNNING)"
2017-11-08T07:01:04Z [INFO] Created container name mapping for task arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 - cs_docServer_container -> ecs-csapps_td-2-csdocServercontainer-faaad4f5948cdde8d101
2017-11-08T07:01:04Z [INFO] Saving state! module="statemanager"
2017-11-08T07:01:04Z [INFO] Created container name mapping for task arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 - cs_tomcat_container -> ecs-csapps_td-2-cstomcatcontainer-da92c589a3da81c2ae01
2017-11-08T07:01:04Z [INFO] Saving state! module="statemanager"
2017-11-08T07:01:05Z [INFO] Created docker container for task arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327: cs_docServer_container -> fa5de8ea4260647093a89b0524a3ed30ca7fd73862644406541ff21b83cd726a
2017-11-08T07:01:05Z [INFO] Redundant container state change for task csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (NONE->RUNNING) Containers: [cs_tomcat_container (PULLED->RUNNING),cs_docServer_container (CREATED->RUNNING),]: cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (CREATED->RUNNING) to CREATED, but already CREATED
2017-11-08T07:01:05Z [INFO] Created docker container for task arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327: cs_tomcat_container -> 8825ec236146a35d08e9aafc4c83b66ced99c609c89fd96974c9279657568d66
2017-11-08T07:01:05Z [INFO] Redundant container state change for task csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (CREATED->RUNNING) Containers: [cs_tomcat_container (CREATED->RUNNING),cs_docServer_container (CREATED->RUNNING),]: cs_tomcat_container(image2YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY) (CREATED->RUNNING) to CREATED, but already CREATED
2017-11-08T07:01:05Z [INFO] Starting container module="TaskEngine" task="csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (CREATED->RUNNING) Containers: [cs_tomcat_container (CREATED->RUNNING),cs_docServer_container (CREATED->RUNNING),]" container="cs_tomcat_container(image2YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY) (CREATED->RUNNING)"
2017-11-08T07:01:05Z [INFO] Starting container module="TaskEngine" task="csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (CREATED->RUNNING) Containers: [cs_tomcat_container (CREATED->RUNNING),cs_docServer_container (CREATED->RUNNING),]" container="cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (CREATED->RUNNING)"
2017-11-08T07:01:05Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 cs_tomcat_container -> RUNNING, Ports [{8080 80 0.0.0.0 0}], Known Sent: NONE
2017-11-08T07:01:05Z [INFO] Redundant container state change for task csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (CREATED->RUNNING) Containers: [cs_tomcat_container (RUNNING->RUNNING),cs_docServer_container (CREATED->RUNNING),]: cs_tomcat_container(image2YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-11-08T07:01:06Z [INFO] Task change event module="TaskEngine" event="{Attachment:<nil> TaskARN:arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 Status:RUNNING Reason: Containers:[] Task:csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (RUNNING->RUNNING) Containers: [cs_tomcat_container (RUNNING->RUNNING),cs_docServer_container (RUNNING->RUNNING),]}"
2017-11-08T07:01:06Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 cs_docServer_container -> RUNNING, Ports [{8080 81 0.0.0.0 0}], Known Sent: NONE
2017-11-08T07:01:06Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 -> RUNNING, Known Sent: NONE] sent: false
2017-11-08T07:01:06Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 -> RUNNING, Known Sent: NONE] sent: false
2017-11-08T07:01:06Z [INFO] Redundant container state change for task csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (RUNNING->RUNNING) Containers: [cs_tomcat_container (RUNNING->RUNNING),cs_docServer_container (RUNNING->RUNNING),]: cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-11-08T07:01:06Z [INFO] Stopping container, container: cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (RUNNING->STOPPED), task: csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (RUNNING->STOPPED) Containers: [cs_tomcat_container (STOPPED->STOPPED),cs_docServer_container (RUNNING->STOPPED),]
2017-11-08T07:01:06Z [INFO] Error retrieving stats for container 8825ec236146a35d08e9aafc4c83b66ced99c609c89fd96974c9279657568d66: context canceled
2017-11-08T07:01:06Z [INFO] Container 8825ec236146a35d08e9aafc4c83b66ced99c609c89fd96974c9279657568d66 is terminal, stopping stats collection
2017-11-08T07:01:06Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 cs_tomcat_container -> STOPPED, Exit 0, , Known Sent: RUNNING
2017-11-08T07:01:06Z [WARN] Error converting stats for container fa5de8ea4260647093a89b0524a3ed30ca7fd73862644406541ff21b83cd726a: Invalid container statistics reported
2017-11-08T07:01:06Z [INFO] Task change event module="TaskEngine" event="{Attachment:<nil> TaskARN:arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 Status:STOPPED Reason: Containers:[] Task:csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (STOPPED->STOPPED) Containers: [cs_tomcat_container (STOPPED->STOPPED),cs_docServer_container (STOPPED->STOPPED),]}"
2017-11-08T07:01:06Z [INFO] Error retrieving stats for container fa5de8ea4260647093a89b0524a3ed30ca7fd73862644406541ff21b83cd726a: context canceled
2017-11-08T07:01:06Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 cs_docServer_container -> STOPPED, Exit 0, , Known Sent: RUNNING
2017-11-08T07:01:06Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 -> STOPPED, Known Sent: RUNNING] sent: false
2017-11-08T07:01:06Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327 -> STOPPED, Known Sent: RUNNING] sent: false
2017-11-08T07:01:06Z [INFO] Error transitioning container module="TaskEngine" task="csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (STOPPED->STOPPED) Containers: [cs_tomcat_container (STOPPED->STOPPED),cs_docServer_container (STOPPED->STOPPED),]" container="cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (STOPPED->STOPPED) - Exit: 0" state="STOPPED" error="Container not running: fa5de8ea4260647093a89b0524a3ed30ca7fd73862644406541ff21b83cd726a"
2017-11-08T07:01:06Z [INFO] Container fa5de8ea4260647093a89b0524a3ed30ca7fd73862644406541ff21b83cd726a is terminal, stopping stats collection
2017-11-08T07:01:06Z [INFO] Redundant container state change for task csapps_td:2 arn:aws:ecs:us-east-2:690021756357:task/0ac25d9a-ddf4-4fe5-9f9b-8dafc1825327, TaskStatus: (STOPPED->STOPPED) Containers: [cs_tomcat_container (STOPPED->STOPPED),cs_docServer_container (STOPPED->STOPPED),]: cs_docServer_container(image1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxx) (STOPPED->STOPPED) - Exit: 0 to STOPPED, but already STOPPED
2017-11-08T07:01:14Z [INFO] Saving state! module="statemanager"
来源：https://stackoverflow.com/questions/47127132/aws-ecs-docker-container-exited-with-code-0