How to clone all projects of a group at once in GitLab?

Question

In my GitLab repository, I have a group with 20 projects. I want to clone all projects at once. Is that possible?

Answer 1

Not really, unless:

• you have a 21st project which references the other 20 as submodules.
  (in which case a clone followed by a git submodule update --init would be enough to get all 20 projects cloned and checked out)

• or you somehow list the projects you have access (GitLab API for projects), and loop on that result to clone each one (meaning that can be scripted, and then executed as "one" command)

---

Since 2015, Jay Gabez mentions in the comments (August 2019) the tool gabrie30/ghorg

ghorg allows you to quickly clone all of an org's or user's repos into a single directory.

Usage:

$ ghorg clone someorg
$ ghorg clone someuser --clone-type=user --protocol=ssh --branch=develop
$ ghorg clone gitlab-org --scm=gitlab --namespace=gitlab-org/security-products
$ ghorg clone --help

---

Also (2020): https://github.com/ezbz/gitlabber

usage: gitlabber [-h] [-t token] [-u url] [--debug] [-p]
                [--print-format {json,yaml,tree}] [-i csv] [-x csv]
                [--version]
                [dest]

Gitlabber - clones or pulls entire groups/projects tree from gitlab

Answer 2

I have written the script to pull the complete code base from gitlab for particular group.

for pag in {1..3} // number of pages projects has span {per page 20 projects so if you have 50 projects loop should be 1..3}
do
curl -s http://gitlink/api/v4/groups/{groupName}/projects?page=$pag > url.txt
grep -o '"ssh_url_to_repo": *"[^"]*"' url.txt | grep -o '"[^"]*"$' | while read -r line ; do
l1=${line%?}
l2=${l1:1}
echo "$l2"
git clone $l2
done
done

Answer 3

Here is another example of a bash script to clone all the repos in a group. The only dependency you need to install is jq (https://stedolan.github.io/jq/). Simply place the script into the directory you want to clone your projects into. Then run it as follows:

./myscript <group name> <private token> <gitlab url>

i.e.

./myscript group1 abc123tyn234 http://yourserver.git.com

Script:

#!/bin/bash
if command -v jq >/dev/null 2>&1; then
  echo "jq parser found";
else
  echo "this script requires the 'jq' json parser (https://stedolan.github.io/jq/).";
  exit 1;
fi

if [ -z "$1" ]
  then
    echo "a group name arg is required"
    exit 1;
fi

if [ -z "$2" ]
  then
    echo "an auth token arg is required. See $3/profile/account"
    exit 1;
fi

if [ -z "$3" ]
  then
    echo "a gitlab URL is required."
    exit 1;
fi

TOKEN="$2";
URL="$3/api/v3"
PREFIX="ssh_url_to_repo";

echo "Cloning all git projects in group $1";

GROUP_ID=$(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups?search=$1 | jq '.[].id')
echo "group id was $GROUP_ID";
curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$GROUP_ID/projects?per_page=100 | jq --arg p "$PREFIX" '.[] | .[$p]' | xargs -L1 git clone

Answer 4

In response to @Kosrat D. Ahmad as I had the same issue (with nested subgroups - mine actually went as much as 5 deep!)

#!/bin/bash
URL="https://mygitlaburl/api/v4"
TOKEN="mytoken"

function check_subgroup {
echo "checking $gid"
if [[ $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid/subgroups/ | jq .[].id -r) != "" ]]; then
  for gid in $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid/subgroups/ | jq .[].id -r)
  do
    check_subgroup
  done
else
  echo $gid >> top_level
fi
}

> top_level #empty file
> repos #empty file
for gid in $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/ | jq .[].id -r)
do
  check_subgroup
done
# This is necessary because there will be duplicates if each group has multiple nested groups. I'm sure there's a more elegant way to do this though!
for gid in $(sort top_level | uniq)
do
  curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid | jq .projects[].http_url_to_repo -r >> repos
done

while read repo; do
  git clone $repo
done <repos

rm top_level
rm repos

Note: I use jq .projects[].http_url_to_repo this can be replaced with .ssh_url_to_repo if you'd prefer.

Alternatively strip out the rm's and look at the files individually to check the output etc.

Admittedly this will clone everything, but you can tweak it however you want.

Resources: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-subgroups

Answer 5

Here's an example in Python 3:

from urllib.request import urlopen
import json
import subprocess, shlex

allProjects     = urlopen("http://[yourServer:port]/api/v4/projects?private_token=[yourPrivateTokenFromUserProfile]&per_page=100000")
allProjectsDict = json.loads(allProjects.read().decode())
for thisProject in allProjectsDict: 
    try:
        thisProjectURL  = thisProject['ssh_url_to_repo']
        command     = shlex.split('git clone %s' % thisProjectURL)
        resultCode  = subprocess.Popen(command)

    except Exception as e:
        print("Error on %s: %s" % (thisProjectURL, e.strerror))

Answer 6

I created a tool for that: https://github.com/ezbz/gitlabber, you can use glob/regex expressions to select groups/subgroups you'd like to clone.

Say your top-level group is called MyGroup and you want to clone all projects under it to ~/GitlabRoot you can use the following command:

    gitlabber -t <personal access token> -u <gitlab url> -i '/MyGroup**' ~/GitlabRoot