How do I enable cloning over SSH for a Gitlab runner?

Question

I am having some trouble cloning large repositories over HTTP on my Windows Gitlab runner. I\'ve tried several methods to do shallow clones or disable clone compression. Sti

Answer 1

I had a similar problem that necessitated the use of cloning via ssh: using the virtualbox executor with very old guest linux OSes. I was able to get around it by doing a few small configuration changes:

1. Create a deploy key for access to the project.

2. Force the user account that will perform the clone to use the deploy key. In my virtualbox case, I modified the ssh configuration for the user that's configured for virtualbox in /etc/gitlab-runnner/config.toml.

~/.ssh/config

Host gitlab.example.com
  Preferredauthentications publickey
  IdentityFile ~/.ssh/deploy-key

3. Configure the runner to perform the clone via ssh in /etc/config.toml.

/etc/config.toml

[[runners]]

  # [...]

  environment = ["GIT_STRATEGY=none"]
  pre_build_script = '''
    # Fetching using ssh (via pre_build_script in config.toml)
    if [ -d "${CI_PROJECT_DIR}" ]; then rm -rf "${CI_PROJECT_DIR}"; fi
    mkdir -p "${CI_PROJECT_DIR}"
    cd "${CI_PROJECT_DIR}"
    git init
    git remote add origin "ssh://git@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
    git fetch origin "${CI_COMMIT_SHA}"
    git reset --hard FETCH_HEAD
  '''

  # [...]

Here's a breakdown of the additions to config.toml:

• The GIT_STRATEGY=none environment variable disables the runner's internal git cloning mechanism. (See the Git Strategy section of the CI/CD reference)
• The pre_build_script performs the actual clone using predefined CI/CD environment variables. In my case, this is a bash script to perform something similar to what a GIT_STRATEGY=fetch might do.
• If pre_build_script is multi-line, the output of the runner will only show the first line. Having a comment as the first line helps add clarity to the runner output.
• pre_clone_script is not used here. It's disabled since the environment has GIT_STRATEGY=none set.

Answer 2

According to:

https://docs.gitlab.com/ee/ci/ssh_keys/README.html

You need to:

1. Create a new SSH key pair with ssh-keygen
2. Add the private key as a Secret Variable to the project
3. Run the ssh-agent during job to load the private key.

Example gitlab_ci.yml:

before_script:
  # Install ssh-agent if not already installed, it is required by Docker.
  # (change apt-get to yum if you use a CentOS-based image)
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'

  # Run ssh-agent (inside the build environment)
  - eval $(ssh-agent -s)

  # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  - ssh-add <(echo "$SSH_PRIVATE_KEY")

  # For Docker builds disable host key checking. Be aware that by adding that
  # you are suspectible to man-in-the-middle attacks.
  # WARNING: Use this only with the Docker executor, if you use it with shell
  # you will overwrite your user's SSH config.
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  # In order to properly check the server's host key, assuming you created the
  # SSH_SERVER_HOSTKEYS variable previously, uncomment the following two lines
  # instead.
  # - mkdir -p ~/.ssh
  # - '[[ -f /.dockerenv ]] && echo "$SSH_SERVER_HOSTKEYS" > ~/.ssh/known_hosts'

Answer 3

As a newcomer to gitlab, I've managed to hack a workaround to this issue as I also haven't found a built-in way to change the default cloning process (although here is a recent comment about how it can be done).

By disabling the automatic cloning process, you can effectively override its behavior completely by simply writing your own cloning process in a before_script. Only for the purposes of example does the below show how to accomplish this for HTTP cloning but could be adapted for ssh cloning (if you're trying to use HTTP cloning you should use the built-in cloning process and the config.toml):

1. Create a new user called "gitlab-runner" and generate their user auth token for later use (or in your case, you would generate ssh keys).

2. Disable cloning process for runner by adding the following variable in either your project or group settings: .../settings/ci_cd

   key: GIT_STRATEGY

   value: none

3. Clone your repo in a before_script such as:

before_script:
  ## clean the working directory
  - BUILD_DIR=/home/gitlab-runner/builds/$RUNNER_TOKEN/0
  - CLONE_DIR="$BUILD_DIR/$CI_PROJECT_PATH"
  - cd $BUILD_DIR
  - rm -rf $CLONE_DIR
  - mkdir -p $CLONE_DIR

  ## clone the project each time (inefficient, consider performing fetch instead if it already exists)
  - git clone http://gitlab-runner:$GITLABRUNNER_USER_AUTH_TOKEN@server:8888/${CI_PROJECT_PATH}.git $CLONE_DIR
  - cd $CLONE_DIR

Note: Here are the relevant variables I also configured in step 2 rather than hard coding them in the script:

• RUNNER_TOKEN: "Runner Token" value listed in the Admin "Runners" menu for the particular runner you are trying to run.
• GITLABRUNNER_USER_AUTH_TOKEN: This is the auth token you generated in step 1.