GitLab to GitHub migration fails due to file size limit of 100.00 MB

问题

I am trying to Migrate one repository from GitLab to GitHub. The repository size is 685.83MB and it consists of few .dat,.csv,.exe,.pkl files which are more than 100MB to 3383.40 MB. it is failing with below errors.

GitLab To GitHub Migration Steps:-
$ git clone --mirror git@your-gitlab-site.com:test/my-repo.git
$ cd ~/my-repo.git
$ git remote set-url --push origin git@github.com:test/my-repo.git
$ git push

Error
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: File Src/project/label/file1.dat is 476.32 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File Src/models/label/file2.dat is 2431.49 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File test/test1/label/model/file3.exe is 1031.94 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File test/test2/usecase/filemarker/file3.csv is 997.02 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File src/msg/sports/model.pkl is 3383.40 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File test/movie/maker/marker.dat is 1373.45 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File project/make/level/project/realmaker.csv is 1594.83 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File src/moderm/network/test.pkl is 111.07 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB

Git LFS/BFG  Method:
$ git clone --mirror gitlab-heavy-repo 
$ cd gitlab-heavy-repo.git 
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.dat' --no-blob-protection
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.exe' --no-blob-protection
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.csv' --no-blob-protection
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.pkl' --no-blob-protection
$ git reflog expire --expire=now --all && git gc --prune=now
$ git lfs install
$ git remote set-url origin git@github.com:some-org/githubheavy-repo.git
$ git push

Even after above process, it fails with same error. it seems Git LFS have 2GB Limitation. So tried to remove the above larger files completely from repository. Followed below method to remove.

1) git clone gitlab-heavy-repo
2) cd gitlab-heavy-repo
3) git filter-branch --force --index-filter "git rm --cached --ignore-unmatch Src/project/label/file1.dat" --prune-empty --tag-name-filter cat -- --all
4) git reflog expire --expire=now --all
5) git gc --prune=now
6) git push origin --force --all
7) git push origin --force --tags
8) rm -rf .git/refs/original/

Repeated the same steps for all the above larger files. But now in Gitlab repository storage size shows - 1.9-GB initially it was only 685.83MB.

Please correct me. Thanks in advance.

回答1:

You could run this to add all files above 100MiB to .gitignore:

find . -size +100M | cat >> .gitignore

Taken from here.

Afterwards, to read files from .gitignore and remove them from repo (without deleting them from disk), run the following command (will work on Linux only):

git ls-files -ci --exclude-standard -z | xargs -0 git rm --cached

Taken from here

Update: If you already have these files committed, you'll need to clean them from commit history. Firstly, confirm that you have all the files from the error message added to your .gitignore.
Then run the following commands for each file to remove it from all previous commits:

git filter-branch --prune-empty -d /dev/shm/scratch \
  --index-filter "git rm --cached -f --ignore-unmatch PATH/TO/FILE" \
  --tag-name-filter cat -- --all

@GregBacon explains this command better in his answer here

Update: If you need to run the command above on a specific folder (recursively), run the following commands instead:

git filter-branch --prune-empty -d /dev/shm/scratch \
  --index-filter "git rm -r --cached -f --ignore-unmatch PATH/TO/FOLDER" \
  --tag-name-filter cat -- --all

Notice the -r (recursive) strategy after git rm in the second line.

来源：https://stackoverflow.com/questions/62382501/gitlab-to-github-migration-fails-due-to-file-size-limit-of-100-00-mb

标签

git

github

gitlab

git-lfs