How to automatically update your docker containers, if base-images are updated

Question

Say I have a trivial container based on the ubuntu:latest. Now there is a security update and ubuntu:latest is updated in the docker repo .

Answer 1

There are a lot of answers here, but none of them suited my needs. I wanted an actual answer to the asker's #1 question. How do I know when an image is updated on hub.docker.com?

The below script can be run daily. On first run, it gets a baseline of the tags and update dates from the HUB registry and saves them locally. From then out, every time it is run it checks the registry for new tags and update dates. Since this changes every time a new image exists, it tells us if the base image has changed. Here is the script:

#!/bin/bash

DATAPATH='/data/docker/updater/data'

if [ ! -d "${DATAPATH}" ]; then
        mkdir "${DATAPATH}";
fi
IMAGES=$(docker ps --format "{{.Image}}")
for IMAGE in $IMAGES; do
        ORIGIMAGE=${IMAGE}
        if [[ "$IMAGE" != *\/* ]]; then
                IMAGE=library/${IMAGE}
        fi
        IMAGE=${IMAGE%%:*}
        echo "Checking ${IMAGE}"
        PARSED=${IMAGE//\//.}
        if [ ! -f "${DATAPATH}/${PARSED}" ]; then
                # File doesn't exist yet, make baseline
                echo "Setting baseline for ${IMAGE}"
                curl -s "https://registry.hub.docker.com/v2/repositories/${IMAGE}/tags/" > "${DATAPATH}/${PARSED}"
        else
                # File does exist, do a compare
                NEW=$(curl -s "https://registry.hub.docker.com/v2/repositories/${IMAGE}/tags/")
                OLD=$(cat "${DATAPATH}/${PARSED}")
                if [[ "${VAR1}" == "${VAR2}" ]]; then
                        echo "Image ${IMAGE} is up to date";
                else
                        echo ${NEW} > "${DATAPATH}/${PARSED}"
                        echo "Image ${IMAGE} needs to be updated";
                        H=`hostname`
                        ssh -i /data/keys/<KEYFILE> <USER>@<REMOTEHOST>.com "{ echo \"MAIL FROM: root@${H}\"; echo \"RCPT TO: <USER>@<EMAILHOST>.com\"; echo \"DATA\"; echo \"Subject: ${H} - ${IMAGE} needs update\"; echo \"\"; echo -e \"\n${IMAGE} needs update.\n\ndocker pull ${ORIGIMAGE}\"; echo \"\"; echo \".\"; echo \"quit\"; sleep 1; } | telnet <SMTPHOST> 25"
                fi

        fi
done;

You will want to alter the DATAPATH variable at the top, and alter the email notification command at the end to suit your needs. For me, I have it SSH into a server on another network where my SMTP is located. But you could easily use the mail command, too.

Now, you also want to check for updated packages inside the containers themselves. This is actually probably more effective than doing a "pull" once your containers are working. Here's the script to pull that off:

#!/bin/bash


function needsUpdates() {
        RESULT=$(docker exec ${1} bash -c ' \
                if [[ -f /etc/apt/sources.list ]]; then \
                grep security /etc/apt/sources.list > /tmp/security.list; \
                apt-get update > /dev/null; \
                apt-get upgrade -oDir::Etc::Sourcelist=/tmp/security.list -s; \
                fi; \
                ')
        RESULT=$(echo $RESULT)
        GOODRESULT="Reading package lists... Building dependency tree... Reading state information... Calculating upgrade... 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded."
        if [[ "${RESULT}" != "" ]] && [[ "${RESULT}" != "${GOODRESULT}" ]]; then
                return 0
        else
                return 1
        fi
}

function sendEmail() {
        echo "Container ${1} needs security updates";
        H=`hostname`
        ssh -i /data/keys/<KEYFILE> <USRER>@<REMOTEHOST>.com "{ echo \"MAIL FROM: root@${H}\"; echo \"RCPT TO: <USER>@<EMAILHOST>.com\"; echo \"DATA\"; echo \"Subject: ${H} - ${1} container needs security update\"; echo \"\"; echo -e \"\n${1} container needs update.\n\n\"; echo -e \"docker exec ${1} bash -c 'grep security /etc/apt/sources.list > /tmp/security.list; apt-get update > /dev/null; apt-get upgrade -oDir::Etc::Sourcelist=/tmp/security.list -s'\n\n\"; echo \"Remove the -s to run the update\"; echo \"\"; echo \".\"; echo \"quit\"; sleep 1; } | telnet <SMTPHOST> 25"
}

CONTAINERS=$(docker ps --format "{{.Names}}")
for CONTAINER in $CONTAINERS; do
        echo "Checking ${CONTAINER}"
        if needsUpdates $CONTAINER; then
                sendEmail $CONTAINER
        fi
done

Answer 2

have you tried this: https://github.com/v2tec/watchtower. it's a simple tool running in docker container watching other containers, if their base image changed, it will pull and redeploy.

Answer 3

You can use Watchtower to watch for updates to the image a container is instantiated from and automatically pull the update and restart the container using the updated image. However, that doesn't solve the problem of rebuilding your own custom images when there's a change to the upstream image it's based on. You could view this as a two-part problem: (1) knowing when an upstream image has been updated, and (2) doing the actual image rebuild. (1) can be solved fairly easily, but (2) depends a lot on your local build environment/practices, so it's probably much harder to create a generalized solution for that.

If you're able to use Docker Hub's automated builds, the whole problem can be solved relatively cleanly using the repository links feature, which lets you trigger a rebuild automatically when a linked repository (probably an upstream one) is updated. You can also configure a webhook to notify you when an automated build occurs. If you want an email or SMS notification, you could connect the webhook to IFTTT Maker. I found the IFTTT user interface to be kind of confusing, but you would configure the Docker webhook to post to https://maker.ifttt.com/trigger/`docker_xyz_image_built`/with/key/`your_key`.

If you need to build locally, you can at least solve the problem of getting notifications when an upstream image is updated by creating a dummy repo in Docker Hub linked to your repo(s) of interest. The sole purpose of the dummy repo would be to trigger a webhook when it gets rebuilt (which implies one of its linked repos was updated). If you're able to receive this webhook, you could even use that to trigger a rebuild on your side.

Answer 4

I had the same issue and thought it can be simply solved by a cron job calling unattended-upgrade daily.

My intention is to have this as an automatic and quick solution to ensure that production container is secure and updated because it can take me sometime to update my images and deploy a new docker image with the latest security updates.

It is also possible to automate the image build and deployment with Github hooks

I've created a basic docker image with that automatically checks and installs security updates daily (can run directly by docker run itech/docker-unattended-upgrade ).

I also came across another different approach to check if the container needs an update.

My complete implementation:

Dockerfile

FROM ubuntu:14.04   

RUN apt-get update \
&& apt-get install -y supervisor unattended-upgrades \
&& rm -rf /var/lib/apt/lists/*

COPY install /install
RUN chmod 755 install
RUN /install

COPY start /start
RUN chmod 755 /start

Helper scripts

install

#!/bin/bash
set -e

cat > /etc/supervisor/conf.d/cron.conf <<EOF
[program:cron]
priority=20
directory=/tmp
command=/usr/sbin/cron -f
user=root
autostart=true
autorestart=true
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
EOF

rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["/start"]

start

#!/bin/bash

set -e

echo "Adding crontab for unattended-upgrade ..."
echo "0 0 * * * root /usr/bin/unattended-upgrade" >> /etc/crontab

# can also use @daily syntax or use /etc/cron.daily

echo "Starting supervisord ..."
exec /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf

Edit

I developed a small tool docker-run that runs as docker container and can be used to update packages inside all or selected running containers, it can also be used to run any arbitrary commands.

Can be easily tested with the following command:

docker run --rm -v /var/run/docker.sock:/tmp/docker.sock itech/docker-run exec

which by default will execute date command in all running containers and display the results. If you pass update instead of exec it will execute apt-get update followed by apt-get upgrade -y in all running containers