How to import git repositories with large files?

Question

Given that GitHub doesn't allow to push files larger than 100 MB, it is not possible to git clone and push a repository with large files into GitHub enterprise. The push fails with a:

remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: File large.zip is 145.00 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB

^{(N.B.: there is a repository importer, but this is only for github.com, and requires public access to your repository)}

Fortunately, GitHub provides support for storage of files larger than 100MB since April 2015. So how to convert a current repository with such large files into a GitHub LFS compatible repository I can push to?

Answer 1

You can now use git lfs migrate built-in command to both assess which files is best to migrate and also do the actual history-rewriting.

See git-lfs migration tutorial for more details.

Answer 2

The easiest way I found was taking advantage of git filter-branch and the BFG Repo-Cleaner by rtyley (I used version 1.12.12):

1. Prerequisite: you need to have git lfs installed

2. Create a new repository on GitHub Enterprise. You'll import your external Git repository to this new repository.

3. Clone the repository you want to migrate to a local folder:

$ git clone --mirror git@oldgithost:repo
$ cd repo.git
# mirror into a local directory

4. Rewrite the history to lfs-track your large files¹:

$ git filter-branch --tree-filter 'git lfs track "*.{zip,jar}"' -- --all
# writes the patterns to lsf-track into .gitattributes

5. Use the BFG to extract the relevant files into Git LFS

$ java -jar ~/usr/bfg-repo-cleaner/bfg-1.12.12.jar --convert-to-git-lfs '*.zip'
$ java -jar ~/usr/bfg-repo-cleaner/bfg-1.12.12.jar --convert-to-git-lfs '*.jar'
# Convert large files (I couldn't find a one-liner for multiple patterns)

6. Push to your GitHub enterprise remote:

$ git push --mirror https://hostname/ghuser/repo.git
# Pushes the mirror to the new GitHub Enterprise repository

7. Delete temporary dir:

$ cd ..
$ rm -rf repo.git

---

Notes

¹ Due to the high I/O, it is recommended to rewrite the history into a temporary directory off-disk with the -d option, e.g. on tmpfs.