Combining multiple git repositories

Question

Let\'s say I\'ve got a setup that look something like

phd/code/
phd/figures/
phd/thesis/

For historical reasons, these all have their own g

Answer 1

This bash script works around the sed tab character issue (on MacOS for example) and the issue of missing files.

export SUBREPO="subrepo"; # <= your subrepository name here
export TABULATOR=`printf '\t'`;
FILTER='git ls-files -s | sed "s#${TABULATOR}#&${SUBREPO}/#" |
  GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
  git update-index --index-info &&
  if [ -f "$GIT_INDEX_FILE.new" ]; then mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE; else echo "git filter skipped missing file: $GIT_INXEX_FILE.new"; fi'

git filter-branch --index-filter "$FILTER" HEAD

This is a combination of miniquark, marius-butuc and ryan's posts. Cheers to them!

Answer 2

I have created a tool that make this task. The method used is similar (internally make some things like --filter-branch) but is more friendly. Is GPL 2.0

http://github.com/geppo12/GitCombineRepo

Answer 3

@MiniQuark solution helped me a lot, but unfortunately it doesn't take into account tags which are in source repositories (At least in my case). Below is my improvement to @MiniQuark answer.

1. First create directory which will contain composed repo and merged repos, create directory for each merged one.

   $ mkdir new_phd
   $ mkdir new_phd/code
   $ mkdir new_phd/figures
   $ mkdir new_phd/thesis
   

2. Do a pull of each repository and fetch all tags. (Presenting instructions only for code sub-directory)

   $ cd new_phd/code
   $ git init
   $ git pull ../../original_phd/code master
   $ git fetch ../../original_phd/code refs/tags/*:refs/tags/*
   

3. (This is improvement to point 2 in MiniQuark answer) Move the content of new_phd/code to new_phd/code/code and add code_ prefeix before each tag

   $ git filter-branch --index-filter 'git ls-files -s | sed "s-\t\"*-&code/-" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' --tag-name-filter 'sed "s-.*-code_&-"' HEAD

4. After doing so there will be twice as many tags as it was before doing filter-branch. Old tags remain in repo and new tags with code_ prefix are added.

   $ git tag
   mytag1
   code_mytag1
   

   Remove old tags manually:

   $ ls .git/refs/tags/* | grep -v "/code_" | xargs rm

   Repeat point 2,3,4 for other subdirectories

5. Now we have structure of directories as in @MiniQuark anwser point 3.

6. Do as in point 4 of MiniQuark anwser, but after doing a pull and before removing .git dir, fetch tags:

   $ git fetch catalog refs/tags/*:refs/tags/*

   Continue..

This is just another solution. Hope it helps someone, it helped me :)

Answer 4

Actually, git-stitch-repo now supports branches and tags, including annotated tags (I found there was a bug which I reported, and it got fixed). What i found useful is with tags. Since tags are attached to commits, and some of the solutions (like Eric Lee's approach) fails to deal with tags. You try to create a branch off an imported tag, and it will undo any git merges/moves and sends you back like the consolidated repository being near identical to the repository that the tag came from. Also, there are issues if you use the same tag across multiple repositories that you 'merged/consolidated'. For example, if you have repo's A ad B, both having tag rel_1.0. You merge repo A and repo B into repo AB. Since rel_1.0 tags are on two different commits (one for A and one for B), which tag will be visible in AB? Either the tag from the imported repo A or from imported repo B, but not both.

git-stitch-repo helps to address that problem by creating rel_1.0-A and rel_1.0-B tags. You may not be able to checkout rel_1.0 tag and expect both, but at least you can see both, and theoretically, you can merge them into a common local branch then create a rel_1.0 tag on that merged branch (assuming you just merge and not change source code). It's better to work with branches, as you can merge like branches from each repo into local branches. (dev-a and dev-b can be merged into a local dev branch which can then be pushed to origin).

Answer 5

git-stitch-repo will process the output of git-fast-export --all --date-order on the git repositories given on the command-line, and create a stream suitable for git-fast-import that will create a new repository containing all the commits in a new commit tree that respects the history of all the source repositories.

Answer 6

I'll throw my solution in here too. It's basically a fairly simple bash script wrapper around git filter-branch. Like other solutions it only migrates master branches and doesn't migrate tags. But the full master commit histories are migrated and it is a short bash script so it should be relatively easy for users to review or tweak.

https://github.com/Oakleon/git-join-repos