How to prevent Dockerfile caching git clone

Question

I have a Dockerfile trying to package and deploy a web app to a container. The code of app fetches from git repository during Docker image building. Here\'s the Dockerfile s

Answer 1

you can also use:

ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache

RUN git reset --hard ~Whatever~

as mentioned here: https://stackoverflow.com/a/58801213/8953378

Answer 2

Issue 1996 is not yet available, but you have the following workaround:

FROM foo
ARG CACHE_DATE=2016-01-01
RUN git clone ...

docker build --build-arg CACHE_DATE=$(date) ....

That would invalidate cache after the ARG CACHE_DATE line for every build.

Or:

ADD http://www.convert-unix-time.com/api?timestamp=now /tmp/bustcache
RUN git pull

That would also invalidate cache after this ADD line.

Similar idea:

Add ARG command to your Dockerfile:

# Dockerfile
# add this and below command will run without cache
ARG CACHEBUST=1

When you need to rebuild with selected cache, run it with --build-arg option

$ docker build -t your-image --build-arg CACHEBUST=$(date +%s) .

then only layer below ARG command in Dockerfile will rebuild.

Answer 3

for anyone who has this problem with Gitlab repositories:

Gitlab has this annoying branch id method when calling their API, the ID will appear under your repository name

# this will copy the last change from your brach and it'll invalidate the cache if there was a new change
ADD "https://gitlab.com/api/v4/projects/${PROJECT_ID}/repository/branches/master?private_token=${GIT_TOKEN}" /tmp/devalidateCache

# the actual clone
RUN git clone --depth=1 https://${GIT_USER}:${GIT_TOKEN}@gitlab.com/${git_file_uri} ${BASE_BUILD_PATH}

Answer 4

Another workaround:

If you use GitHub (or gitlab or bitbucket too most likely) you can ADD the GitHub API's representation of your repo to a dummy location.

ADD https://api.github.com/repos/$USER/$REPO/git/refs/heads/$BRANCH version.json
RUN git clone -b $BRANCH https://github.com/$USER/$REPO.git $GIT_HOME/

The API call will return different results when the head changes, invalidating the docker cache.

If you're dealing with private repos you can use github's x-oauth-basic authentication scheme with a personal access token like so:

ADD https://$ACCESS_TOKEN:x-oauth-basic@api.github.com/repos/$USER/$REPO/git/refs/heads/$BRANCH version.json

(thx @captnolimar for a suggested edit to clarify authentication)

Answer 5

For github private repos, you could also pass in your username and password:

RUN git clone -b$BRANCH https://$USER:$PASSWORD@github.com/$USER/$REPO.git $GIT_HOME/

Answer 6

If you use github you can use github API to not cache specific RUN command. You need to have jq installed to parse JSON: apt-get install -y jq

Example:

docker build --build-arg SHA=$(curl -s 'https://api.github.com/repos/Tencent/mars/commits' | jq -r '.[0].sha') -t imageName .

In Dockerfile (ARG command should be right before RUN):

ARG SHA=LATEST
RUN SHA=${SHA} \
    git clone https://github.com/Tencent/mars.git

or if you don't want to install jq

SHA=$(curl -s 'https://api.github.com/repos/Tencent/mars/commits' | grep sha | head -1)

If repository has new commits, git clone will be executed.