How to setup docker to use cache from registry on every build step

Question

I have two servers with docker and one server with my private registry.

I built Dockerfile on the first machine; then I pushed the image to the registry.

Is it

Answer 1

For Docker versions >= 19.03, you can use the new BuildKit features to avoid having to pull the remote image before building. Activate BuildKit by setting the DOCKER_BUILDKIT enviroment variable, then turn on inline caching to store the build cache along with your image in your registry using the BUILDKIT_INLINE_CACHE build argument, as follows:

export DOCKER_BUILDKIT=1
docker build -t registry/imagename:tag --cache-from registry/imagename:tag --build-arg BUILDKIT_INLINE_CACHE=1 .
docker push registry/imagename:tag

Thanks to Nicholas Dille for this solution.

Answer 2

for docker > 1.10, I found something on this issue: https://github.com/docker/docker/issues/20316#issuecomment-221289631

Given this Dockerfile

FROM busybox
RUN mkdir this-is-a-test
RUN echo "hello world"

run docker build -t caching-test .

Then we can see the layers comprising the image with docker history caching-test

3e4a484f0e67        About an hour ago   /bin/sh -c echo "Hello world!"                  0 B                 
6258cdec0c4b        About an hour ago   /bin/sh -c mkdir this-is-a-test                 0 B                 
47bcc53f74dc        9 weeks ago         /bin/sh -c #(nop) CMD ["sh"]                    0 B                 
<missing>           9 weeks ago         /bin/sh -c #(nop) ADD file:47ca6e777c36a4cfff   1.113 MB  

The change to save/load in 1.11 preserves the relationship between parent and child layers, but only when they are saved via docker save together. We can see the parent of the final test image by running docker inspect test | grep Parent.

$ docker inspect caching-test | grep Parent
"Parent": "sha256:6258cdec0c4bef5e5627f301b541555883e6c4b385d0798a7763cb191168ce09", 

This is the second-to-top layer from our Docker history output.

In order to recreate the cache using save and load, you need to save out all of the images and layers that are referenced as parents. In practice this typically means that you need to save each layer, as well as the FROM image, in the same command.

docker save caching-test 6258cdec0c4b busybox > caching-test.tar -- note that we can also give the layer names instead of IDs to the save command.

Let's purge everything and then reload the image from the tar file. docker rmi $(docker images -q). Confirm that no images exist.

Then run docker load -i caching-test.tar. If you look at the images, you'll see busybox, and then caching-test. Running docker history caching-test will show you the exact same output as when the image was initially built. This is because the parent/child relationships were preserved via save and load. You can even run docker inspect caching-test | grep Parent and see the exact same ID given as the parent layer.

And running a rebuild of the same Dockerfile will show you that the cache is being used.

Sending build context to Docker daemon 5.391 MB
Step 1 : FROM busybox
 ---> 47bcc53f74dc
Step 2 : RUN mkdir this-is-a-test
 ---> Using cache
 ---> 6258cdec0c4b
Step 3 : RUN echo "hello world"
 ---> Using cache
 ---> 3e4a484f0e67
Successfully built 3e4a484f0e67

EDIT: Below this works only Before docker 1.10

On the second machine you can docker pull theimagefromthefirstdockerfileontheregistry before building the new one.

That way you are sure that every layer is present on the second machine.

The docker-engine doesnt query the registry each time a layer is built (it doesn't even knows it), it would be too slow/heavy so I dont think there is another way.

Answer 3

Note: issue 20316 ("Pulling build cache") has been closed because PR 26839 ("Implement build cache based on history array") has been merged.

It allows for instance to specify in --cache-from the image from a previous CI build.

Adds capability to specify images used as a cache source on build. These images do not need to have local parent chain and can be pulled from other registries. User needs to make sure to only use trusted images as sources.

Usage:

docker pull myimage:v1.0
docker build --cache-from myimage:v1.0 -t myimage:v1.1 .

See merge commit 7944480, for docker 1.13 (January 2017).

As commented by javipolo:

In case someone is going nuts with reusing layers as I did, the "trick" is to pass to --cache-from the image you are rebuilding (and have it pulled already) and ALSO the image that it uses as base in the FROM.

Example:
Dockerfile for image custom-gource:0.1

FROM base_image:2.2.1
RUN apt-get update && apt-get install gource
COPY myscript.sh /myscript.sh

In order to rebuild in other host without doing the apt-get again, you'll need to:

docker pull custom-gource:0.1
docker build --cache-from=base_image:2.2.1,custom-gource:0.1 . -t custom-gource:0.2

It might seem too obvious but I've been struggling long time with this until I got that you need to include the base image too