Docker container with status “Dead” after consul healthcheck runs

Question

I am using consul\'s healthcheck feature, and I keep getting these these \"dead\" containers:

CONTAINER ID  IMAGE                   COMMAND              CREATED         


        

Answer 1

Update March 2016: issue 9665 has just been closed by PR 21107 (for docker 1.11 possibly)
That should help avoid the "Driver aufs failed to remove root filesystem", "device or resource busy" problem.

---

Original answer May 2015

Dead is one if the container states, which is tested by Container.Start()

if container.removalInProgress || container.Dead {
        return fmt.Errorf("Container is marked for removal and cannot be started.")
}

It is set Dead when stopping fails, in order to prevent that container to be restarting.

Amongst the possible cause of failure, see container.Kill().
It means kill -15 and kill -9 are both failing.

// 1. Send a SIGTERM
if err := container.killPossiblyDeadProcess(15); err != nil {
    logrus.Infof("Failed to send SIGTERM to the process, force killing")
    if err := container.killPossiblyDeadProcess(9); err != nil {

That usually mean, as the OP mention, a busy device or resource, preventing the process to be killed.

Answer 2

There are a lot of bugs caused by EBUSY, in particular when devicemapper is used.

There is a tracker bug for all of the EBUSY related issues. see https://github.com/docker/docker/issues/5684#issuecomment-69052334