Docker image format

Question

I would like to build a Docker image without docker iself. I have looked at Packer, but it requiers to have Docker installed on the builder host.

I have looked at the Docker Registry API documentation but this information doesn't appear to be there.

I guess that the image is simply a tarball, but I would like to see a complete specification of the format, i.e. what exect format is required and whethere there are any metadata files required. I could attempt downloading an image from the registry and look what's inside, but there is no information on how to fetch the image itself.

The idea of my project is to implement a script that creates an image from atefacts I have compiled and uploads it to the registry. I would like to use OpenEmbedded for this purpose, essentially this would be an extention to Bitbake.

Answer 1

The Docker image format is specified here: https://github.com/docker/docker/blob/master/image/spec/v1.md

The simplest possible image is a tar file containing the following:

repositories
uniqid/VERSION
uniqid/json
uniqid/layer.tar

Where VERSION contains 1.0, layer.tar contains the chroot contents and json/repositories are JSON files as specified in the spec above.

The resulting tar can be loaded into docker via docker load < image.tar

Answer 2

After reading James Coyle's blog, I figured that docker save and docker load commands are what I need.

> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
progrium/consul     latest              e9fe5db22401        11 days ago         25.81 MB
> docker save e9fe5db22401 | tar x
> ls e9fe5db22401*
VERSION  json  layer.tar

The VERSION file contains only 1.0, and json contains quite a lot of information:

{
  "id": "e9fe5db224015ddfa5ee9dbe43b414ecee1f3108fb6ed91add11d2f506beabff",
  "parent": "68f9e4929a4152df9b79d0a44eeda042b5555fbd30a36f98ab425780c8d692eb",
  "created": "2014-08-20T17:54:30.98176344Z",
  "container": "3878e7e9b9935b7a1988cb3ebe9cd45150ea4b09768fc1af54e79b224bf35f26",
  "container_config": {
    "Hostname": "7f17ad58b5b8",
    "Domainname": "",
    "User": "",
    "Memory": 0,
    "MemorySwap": 0,
    "CpuShares": 0,
    "Cpuset": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "PortSpecs": null,
    "ExposedPorts": {
      "53/udp": {},
      "8300/tcp": {},
      "8301/tcp": {},
      "8301/udp": {},
      "8302/tcp": {},
      "8302/udp": {},
      "8400/tcp": {},
      "8500/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "HOME=/",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "SHELL=/bin/bash"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) CMD []"
    ],
    "Image": "68f9e4929a4152df9b79d0a44eeda042b5555fbd30a36f98ab425780c8d692eb",
    "Volumes": {
      "/data": {}
    },
    "WorkingDir": "",
    "Entrypoint": [
      "/bin/start"
    ],
    "NetworkDisabled": false,
    "OnBuild": [
      "ADD ./config /config/"
    ]
  },
  "docker_version": "1.1.2",
  "author": "Jeff Lindsay <progrium@gmail.com>",
  "config": {
    "Hostname": "7f17ad58b5b8",
    "Domainname": "",
    "User": "",
    "Memory": 0,
    "MemorySwap": 0,
    "CpuShares": 0,
    "Cpuset": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "PortSpecs": null,
    "ExposedPorts": {
      "53/udp": {},
      "8300/tcp": {},
      "8301/tcp": {},
      "8301/udp": {},
      "8302/tcp": {},
      "8302/udp": {},
      "8400/tcp": {},
      "8500/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "HOME=/",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "SHELL=/bin/bash"
    ],
    "Cmd": [],
    "Image": "68f9e4929a4152df9b79d0a44eeda042b5555fbd30a36f98ab425780c8d692eb",
    "Volumes": {
      "/data": {}
    },
    "WorkingDir": "",
    "Entrypoint": [
      "/bin/start"
    ],
    "NetworkDisabled": false,
    "OnBuild": [
      "ADD ./config /config/"
    ]
  },
  "architecture": "amd64",
  "os": "linux",
  "Size": 0
}

The layer.tar file appears to be empty. So inspected the parent, and the grandparent, both contained no file in their layer.tar files.

So assuming that 4.0K is the standard size for an empty tarball:

 for layer in $(du -hs */layer.tar | grep -v 4.0K | cut -f2)
 do (echo $layer:;tar tvf $layer)
 done

To see that these contain simple incremental changes to the filesystem.

So one conclusion is that it's probably best to just use Docker to build the image and push it the registry, just as Packer does.

The way to build an image from scratch is described in the docs.

It turns out that docker import - scratch doesn't care about what's in the tarball. I simply assumes that is the rootfs.

> touch foo
> tar c foo | docker import - scratch
02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab
> docker save 02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab | tar x
> ls 02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab/
VERSION  json  layer.tar
> tar tvf 02bb6cd70aa2c9fbaba37c8031c7412272d804d50b2ec608e14db054fc0b9fab/layer.tar    
drwxr-xr-x 0/0               0 2014-09-01 13:46 ./
-rw-r--r-- 500/500           0 2014-09-01 13:46 foo

In terms of OpenEmbedded integration, it's probably best to build the rootfs tarball, which is something Yocto provides out of the box, and use the official Python library to import the rootfs tarball with import_image(src='rootfs.tar', repository='scratch') and then push it private registry method.

This is not the most elegant solution, but that's how it would have to work at the moment. Otherwise one probably can just manage and deploy rootfs revisions in their own way, and just use docker import on the target host, which still won't be a nice fit, but is somewhat simple.