Can't run Docker container due device mapper error

Question

I just can\'t create and run new containers in Docker anymore. But in the same time a can run previously created containers.

When I try to do something like this:<

Answer 1

The following is for a Fedora/RHEL system, so you'll need to adjust for Debian...

# systemctl stop docker.service
# thin_check /var/lib/docker/devicemapper/devicemapper/metadata

If there were no errors then proceed with:

# thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata
# systemctl start docker.service
# docker run --name=fpm-5.3 debian:jessie

---

To install thin_check command, run:

# apt-get install -y thin-provisioning-tools

Answer 2

In our case just need to start and stop the service.

Depending on your system you can run:

service docker stop
service docker start

or:

sudo systemctl stop docker.service
sudo systemctl start docker.service

or:

sudo /etc/init.d/docker restart

Answer 3

When the docker partition filled and docker would no longer start after reboot, I encountered this:

# thin_check /var/lib/docker/devicemapper/devicemapper/metadata
examining superblock
examining devices tree
  missing devices: [0, -]
    bad checksum in btree node
examining mapping tree
  thin device 72 is missing mappings [137494, 137594]
    bad checksum in btree node
  thin device 72 is missing mappings [137721, -]
    bad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree node

I was able to repair with this procedure:

# thin_dump -r /var/lib/docker/devicemapper/devicemapper/metadata -o /tmp/metadata.xml
# thin_restore -i /tmp/metadata.xml -o /var/lib/docker/devicemapper/devicemapper/metadata

Answer 4

I've been fighting this issue with Debian 8.2. I had other problems because I run a 4.3.3 Kernel (default is 3.16) with grsec.

Despite the GRSEC issues (mount & chmod denied) I was able to run docker and create some image and container.

Then, I would reboot and docker would just spit out the error. I ran thin_check and what I found was this:

• metadata was fine (I don't use a thin pool right now since docker keeps on telling me that my thin pool isn't a thin pool...)
• data was corrupted.

Tried to repair it but thin_restore crashes.

I realized that: docker daemon ... was working but can't be stopped with systemctl stop docker.service. It says that the service is stopped but the daemon is still in memory (ps -elf | grep docker)

To fix the problem I had to change the DOCKER_STORAGE_OPTIONS in /etc/default/docker

rm -rf /var/lib/docker
reboot

At boot time, the service starts. docker info

Shows the info as expected. Built an image. Rebooted, the service starts fine again. I think that basically the docker daemon can't be stopped and killing it with a:

kill <pid>

Causes the data file to be corrupted and therefore the checksum won't match.

Bottom line is don't mix and match docker.service and docker daemon. At least on Debian/Ubuntu.

Answer 5

I had the same problem and was not able to fix it. I found something promising at: http://grokbase.com/t/gg/docker-user/1563fzdtm7/docker-docker-runs-out-of-space-when-trying-to-create-a-new-image 'The default docker storage driver allocates 10GB storage blocks for your images. Move to overlayfs and avoid this entirely. In the command that starts your docker daemon just add "-s overlay" '

this fixed my problem.