How to automatically update your docker containers, if base-images are updated

Question

Say I have a trivial container based on the ubuntu:latest. Now there is a security update and ubuntu:latest is updated in the docker repo .

Answer 1

UPDATE: Use Dependabot - https://dependabot.com/docker/

BLUF: finding the right insertion point for monitoring changes to a container is the challenge. It would be great if DockerHub would solve this. (Repository Links have been mentioned but note when setting them up on DockerHub - "Trigger a build in this repository whenever the base image is updated on Docker Hub. Only works for non-official images.")

While trying to solve this myself I saw several recommendations for webhooks so I wanted to elaborate on a couple of solutions I have used.

1. Use microbadger.com to track changes in a container and use it's notification webhook feature to trigger an action. I set this up with zapier.com (but you can use any customizable webhook service) to create a new issue in my github repository that uses Alpine as a base image.

   • Pros: You can review the changes reported by microbadger in github before taking action.
   • Cons: Microbadger doesn't let you track a specific tag. Looks like it only tracks 'latest'.

2. Track the RSS feed for git commits to an upstream container. ex. https://github.com/gliderlabs/docker-alpine/commits/rootfs/library-3.8/x86_64. I used zapier.com to monitor this feed and to trigger an automatic build of my container in Travis-CI anytime something is committed. This is a little extreme but you can change the trigger to do other things like open an issue in your git repository for manual intervention.

   • Pros: Closer to an automated pipline. The Travis-CI build just checks to see if your container has issues with whatever was committed to the base image repository. It's up to you if your CI service takes any further action.
   • Cons: Tracking the commit feed isn't perfect. Lots of things get committed to the repository that don't affect the build of the base image. Doesn't take in to account any issues with frequency/number of commits and any API throttling.

Answer 2

Above Answers are also correct

There are two Approach

1. Use webhooks
2. Run script for every specific minute to get fresh pull of docker images

I am just sharing script may be it will helpful for you! You can use it with cronjob, I tried succesfully on OSX

#!/bin/bash
##You can use below commented line for setting cron tab for running cron job and to store its O/P in one .txt file  
#* * * * * /usr/bin/sudo -u admin -i bash -c /Users/Swapnil/Documents/checkimg.sh > /Users/Swapnil/Documents/cron_output.log 2>&1
# Example for the Docker Hub V2 API
# Returns all images and tags associated with a Docker Hub organization account.
# Requires 'jq': https://stedolan.github.io/jq/

# set username, password, and organization
# Filepath where your docker-compose file is present
FILEPATH="/Users/Swapnil/Documents/lamp-alpine"
# Your Docker hub user name
UNAME="ur username"
# Your Docker hub user password
UPASS="ur pwd"
# e.g organisation_name/image_name:image_tag
ORG="ur org name"
IMGNAME="ur img name"
IMGTAG="ur img tag"
# Container name
CONTNAME="ur container name"
# Expected built mins
BUILDMINS="5"
#Generally cronjob frequency
CHECKTIME="5"
NETWORKNAME="${IMGNAME}_private-network"
#After Image pulling, need to bring up all docker services?
DO_DOCKER_COMPOSE_UP=true
# -------
echo "Eecuting Script @ date and time in YmdHMS: $(date +%Y%m%d%H%M%S)"
set -e
PIDFILE=/Users/Swapnil/Documents/$IMGNAME/forever.pid
if [ -f $PIDFILE ]
then
  PID=$(cat $PIDFILE)
  ps -p $PID > /dev/null 2>&1
  if [ $? -eq 0 ]
  then
    echo "Process already running"
    exit 1
  else
    ## Process not found assume not running
    echo $$
    echo $$ > $PIDFILE
    if [ $? -ne 0 ]
    then
      echo "Could not create PID file"
      exit 1
    fi
  fi
else
  echo $$ > $PIDFILE
  if [ $? -ne 0 ]
  then
    echo "Could not create PID file"
    exit 1
  fi
fi

# Check Docker is running or not; If not runing then exit
if docker info|grep Containers ; then
    echo "Docker is running"
else
    echo "Docker is not running"
    rm $PIDFILE
    exit 1
fi

# Check Container is running or not; and set variable
CONT_INFO=$(docker ps -f "name=$CONTNAME" --format "{{.Names}}")
if [ "$CONT_INFO" = "$CONTNAME" ]; then
    echo "Container is running"
    IS_CONTAINER_RUNNING=true
else
    echo "Container is not running"
    IS_CONTAINER_RUNNING=false
fi


# get token
echo "Retrieving token ..."
TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'${UNAME}'", "password": "'${UPASS}'"}' https://hub.docker.com/v2/users/login/ | jq -r .token)

# get list of repositories
echo "Retrieving repository list ..."
REPO_LIST=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/?page_size=100 | jq -r '.results|.[]|.name')

# output images & tags
echo "Images and tags for organization: ${ORG}"
echo
for i in ${REPO_LIST}
do
  echo "${i}:"
  # tags
  IMAGE_TAGS=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${i}/tags/?page_size=100 | jq -r '.results|.[]|.name')
  for j in ${IMAGE_TAGS}
  do
    echo "  - ${j}"
  done
  #echo
done

# Check Perticular image is the latest or not
#imm=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${IMGNAME}/tags/?page_size=100)
echo "-----------------"
echo "Last built date details about Image ${IMGNAME} : ${IMGTAG} for organization: ${ORG}"
IMAGE_UPDATED_DATE=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${IMGNAME}/tags/?page_size=100 | jq -r '.results|.[]|select(.name | contains("'${IMGTAG}'")).last_updated')
echo "On Docker Hub IMAGE_UPDATED_DATE---$IMAGE_UPDATED_DATE"
echo "-----------------"

IMAGE_CREATED_DATE=$(docker image inspect ${ORG}/${IMGNAME}:${IMGTAG} | jq -r '.[]|.Created')
echo "Locally IMAGE_CREATED_DATE---$IMAGE_CREATED_DATE"

updatedDate=$(date -jf '%Y-%m-%dT%H:%M' "${IMAGE_UPDATED_DATE:0:16}" +%Y%m%d%H%M%S) 
createdDate=$(date -jf '%Y-%m-%dT%H:%M' "${IMAGE_CREATED_DATE:0:16}" +%Y%m%d%H%M%S)
currentDate=$(date +%Y%m%d%H%M%S)

start_date=$(date -jf "%Y%m%d%H%M%S" "$currentDate" "+%s")
end_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
updiffMins=$(( ($start_date - $end_date) / (60) ))
if [[ "$updiffMins" -lt $(($CHECKTIME+1)) ]]; then
        if [ ! -d "${FILEPATH}" ]; then
            mkdir "${FILEPATH}";
        fi
        cd "${FILEPATH}"
        pwd
        echo "updatedDate---$updatedDate" > "ScriptOutput_${currentDate}.txt"
        echo "createdDate---$createdDate" >> "ScriptOutput_${currentDate}.txt"
        echo "currentDate---$currentDate" >> "ScriptOutput_${currentDate}.txt"
        echo "Found after regular checking time -> Docker hub's latest updated image is new; Diff ${updiffMins} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Script is checking for latest updates after every ${CHECKTIME} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Fetching all new"
        echo "---------------------------"
        if $IS_CONTAINER_RUNNING ; then
            echo "Container is running"         
        else
            docker-compose down
            echo "Container stopped and removed; Network removed" >> "ScriptOutput_${currentDate}.txt"
        fi
        echo "Image_Created_Date=$currentDate" > ".env"
        echo "ORG=$ORG" >> ".env"
        echo "IMGNAME=$IMGNAME" >> ".env"
        echo "IMGTAG=$IMGTAG" >> ".env"
        echo "CONTNAME=$CONTNAME" >> ".env"
        echo "NETWORKNAME=$NETWORKNAME" >> ".env"
        docker-compose build --no-cache
        echo "Docker Compose built" >> "ScriptOutput_${currentDate}.txt"
        if $DO_DOCKER_COMPOSE_UP ; then
            docker-compose up -d
            echo "Docker services are up now, checked in" >> "ScriptOutput_${currentDate}.txt"  
        else
            echo "Docker services are down, checked in" >> "ScriptOutput_${currentDate}.txt"
        fi
elif [[ "$updatedDate" -gt "$createdDate" ]]; then 
    echo "Updated is latest"
    start_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
    end_date=$(date -jf "%Y%m%d%H%M%S" "$createdDate" "+%s")
    diffMins=$(( ($start_date - $end_date) / (60) ))
    if [[ "$BUILDMINS" -lt "$diffMins" ]]; then
        if [ ! -d "${FILEPATH}" ]; then
            mkdir "${FILEPATH}";
        fi
        cd "${FILEPATH}"
        pwd
        echo "updatedDate---$updatedDate" > "ScriptOutput_${currentDate}.txt"
        echo "createdDate---$createdDate" >> "ScriptOutput_${currentDate}.txt"
        echo "currentDate---$currentDate" >> "ScriptOutput_${currentDate}.txt"
        echo "Found after comparing times -> Docker hub's latest updated image is new; Diff ${diffMins} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Actual image built time is less i.e. ${diffMins} mins than MAX expexted BUILD TIME i.e. ${BUILDMINS} mins" >> "ScriptOutput_${currentDate}.txt"
        echo "Fetching all new" >> "ScriptOutput_${currentDate}.txt"
        echo "-----------------------------"
        if $IS_CONTAINER_RUNNING ; then
            echo "Container is running"         
        else
            docker-compose down
            echo "Container stopped and removed; Network removed" >> "ScriptOutput_${currentDate}.txt"
        fi
        echo "Image_Created_Date=$currentDate" > ".env"
        echo "ORG=$ORG" >> ".env"
        echo "IMGNAME=$IMGNAME" >> ".env"
        echo "IMGTAG=$IMGTAG" >> ".env"
        echo "CONTNAME=$CONTNAME" >> ".env"
        echo "NETWORKNAME=$NETWORKNAME" >> ".env"
        docker-compose build --no-cache
        echo "Docker Compose built" >> "ScriptOutput_${currentDate}.txt"
        if $DO_DOCKER_COMPOSE_UP ; then
            docker-compose up -d
            echo "Docker services are up now" >> "ScriptOutput_${currentDate}.txt"  
        else
            echo "Docker services are down" >> "ScriptOutput_${currentDate}.txt"
        fi
    elif [[ "$BUILDMINS" -gt "$diffMins" ]]; then
        echo "Docker hub's latest updated image is NOT new; Diff ${diffMins} mins"
        echo "Docker images not fetched"
    else
        echo "Docker hub's latest updated image is NOT new; Diff ${diffMins} mins"
        echo "Docker images not fetched"
    fi
elif [[ "$createdDate" -gt "$updatedDate" ]]; then 
    echo "Created is latest"
    start_date=$(date -jf "%Y%m%d%H%M%S" "$createdDate" "+%s")
    end_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
    echo "Docker hub has older docker image than local; Older than $(( ($start_date - $end_date) / (60) ))mins"
fi
echo 
echo "------------end---------------"
rm $PIDFILE

Here is my docker-compose file

version:  "3.2"
services:
  lamp-alpine:
    build:
      context: .
    container_name: "${CONTNAME}"
    image: "${ORG}/${IMGNAME}:${IMGTAG}"
    ports:
      - "127.0.0.1:80:80"
    networks:
      - private-network 

networks:
  private-network:
    driver: bridge

Answer 3

Premise to my answer:

1. Containers are run with tags.
2. The same tag can be pointed to different image UUID as we please/ feel appropriate.
3. Updates done to an image can be committed to a new image layer

Approach

1. Build all the containers in the first place with a security-patch update script
2. Build an automated process for the following
   • Run an existing image to new container with security patch script as the command
   • Commit changes to the image as
     • existing tag -> followed by restarting the containers one by one
     • new version tag -> replace few containers with new tag -> validate -> move all containers to new tag

Additionally, the base image can be upgraded/ the container with a complete new base image can be built at regular intervals, as the maintainer feels necessary

Advantages

1. We are preserving the old version of the image while creating the new security patched image, hence we can rollback to previous running image if necessary
2. We are preserving the docker cache, hence less network transfer (only the changed layer gets on the wire)
3. The upgrade process can be validated in staging before moving to prod
4. This can be a controlled process, hence the security patches only when necessary/ deemed important can be pushed.

Answer 4

Another approach could be to assume that your base image gets behind quite quickly (and that's very likely to happen), and force another image build of your application periodically (e.g. every week) and then re-deploy it if it has changed.

As far as I can tell, popular base images like the official Debian or Java update their tags to cater for security fixes, so tags are not immutable (if you want a stronger guarantee of that you need to use the reference [image:@digest], available in more recent Docker versions). Therefore, if you were to build your image with docker build --pull, then your application should get the latest and greatest of the base image tag you're referencing.

Since mutable tags can be confusing, it's best to increment the version number of your application every time you do this so that at least on your side things are cleaner.

So I'm not sure that the script suggested in one of the previous answers does the job, since it doesn't rebuild you application's image - it just updates the base image tag and then it restarts the container, but the new container still references the old base image hash.

I wouldn't advocate for running cron-type jobs in containers (or any other processes, unless really necessary) as this goes against the mantra of running only one process per container (there are various arguments about why this is better, so I'm not going to go into it here).

Answer 5

You would not know your container is behind without running docker pull. Then you'd need to rebuild or recompose your image.

docker pull image:tag
docker-compose -f docker-compose.yml -f production.yml up -d --build

The commands can be put in a script along with anything else necessary to complete the upgrade, although a proper container would not need anything additional.

Answer 6

One of the ways to do it is to drive this through your CI/CD systems. Once your parent image is built, have something that scans your git repos for images using that parent. If found, you'd then send a pull request to bump to new versions of the image. The pull request, if all tests pass, would be merged and you'd have a new child image based on updated parent. An example of a tool that takes this approach can be found here: https://engineering.salesforce.com/open-sourcing-dockerfile-image-update-6400121c1a75 .

If you don't control your parent image, as would be the case if you are depending on the official ubuntu image, you can write some tooling that detects changes in the parent image tag or checksum(not the same thing, tags are mutable) and invoke children image builds accordingly.