Docker - Cannot remove dead container

Question

I am unable to remove the dead container, it appears again after i restart the Docker service.

docker ps -a
CONTAINER ID         STATUS          
11667ef1623         


        

Answer 1

I had the following error when removing a dead container (docker 17.06.1-ce on CentOS 7):

Error response from daemon: driver "overlay" failed to remove root filesystem for <some-id>: 
remove /var/lib/docker/overlay/<some-id>/merged: device or resource busy

Here is how I fixed it:

1. Check which other processes are also using docker resources

$ grep docker /proc/*/mountinfo

which outputs something like this, where the number after /proc/ is the pid:

/proc/10001/mountinfo:179...
/proc/10002/mountinfo:149...
/proc/12345/mountinfo:159 149 0:36 / /var/lib/docker/overlay/...

2. Check the process name of the above pid

$ ps -p 10001 -o comm=
dockerd
$ ps -p 10002 -o comm=
docker-containe
$ ps -p 12345 -o comm=
nginx   <<<-- This is suspicious!!!

So, nginx with pid 12345 seems to also be using /var/lib/docker/overlay/..., which is why we cannot remove the related container and get the device or resource busy error. (See here for a discussion on how nginx shares the same mount namespace with docker containers thus prevents its deletion.)

3. Stop nginx and then I can remove the container successfully.

$ sudo service nginx stop
$ docker rm <container-id>

Answer 2

In my case, I had to remove it with

rm -r /var/lib/docker/containers/<container-id>/

and it worked. Maybe that's how you solve it in docker version ~19. My docker version was 19.03.12,