Why do excluded files keep reappearing in my git sparse checkout?

Question

I use the GCC git mirror and because I only use the C and C++ front ends I use git's sparse checkout feature to exclude the hundreds of files I don't need:

$ git config core.sparseCheckout
true
$ cat .git/info/sparse-checkout 
/*
!gnattools/
!libada/
!libgfortran/
!libgo/
!libjava/
!libobjc/
!libquadmath/
!gcc/ada/
!gcc/fortran/
!gcc/go/
!gcc/java/
!gcc/objc/
!gcc/objcp/
!gcc/testsuite/ada/
!gcc/testsuite/gfortran.dg/
!gcc/testsuite/gfortran.fortran-torture/
!gcc/testsuite/gnat.dg/
!gcc/testsuite/go.dg/
!gcc/testsuite/go.go-torture/
!gcc/testsuite/go.test/
!gcc/testsuite/objc/
!gcc/testsuite/objc.dg/
!gcc/testsuite/obj-c++.dg/
!gcc/testsuite/objc-obj-c++-shared/

This works for a while, but then now and then I notice that some of those excluded files have returned, sometimes lots of them:

$ ls gnattools/
ChangeLog  configure  configure.ac  Makefile.in
$ ls  gcc/fortran/ | wc -l 
86

I'm not sure exactly when the files reappear, I do a lot of switching to different branches (both remote-tracking and local) and it's a very busy repo so there are new changes to pull frequently.

As a relative newbie to git I don't know how to "reset" my work tree to get rid of those files again.

As an experiment, I tried disabling sparse checkout and pulling, thinking I could enable sparseCheckout again afterwards to update the tree somehow, but that didn't work very well:

$ git config core.sparseCheckout false
$ git config core.sparseCheckout 
false
$ git pull
remote: Counting objects: 276, done.
remote: Compressing objects: 100% (115/115), done.
remote: Total 117 (delta 98), reused 0 (delta 0)
Receiving objects: 100% (117/117), 64.05 KiB, done.
Resolving deltas: 100% (98/98), completed with 64 local objects.
From git://gcc.gnu.org/git/gcc
   7618909..0984ea0  gcc-4_5-branch -> origin/gcc-4_5-branch
   b96fd63..bb95412  gcc-4_6-branch -> origin/gcc-4_6-branch
   d2cdd74..2e8ef12  gcc-4_7-branch -> origin/gcc-4_7-branch
   c62ec2b..fd9cb2c  master     -> origin/master
   2e2713b..29daec8  melt-branch -> origin/melt-branch
   c62ec2b..fd9cb2c  trunk      -> origin/trunk
Updating c62ec2b..fd9cb2c
error: Your local changes to the following files would be overwritten by merge:
        gcc/fortran/ChangeLog
        gcc/fortran/iresolve.c
        libgfortran/ChangeLog
        libgfortran/io/intrinsics.c
Please, commit your changes or stash them before you can merge.
Aborting

So apparently I've got local modifications to files I never asked for and AFAIK have never touched!

But git status doesn't show those changes:

$ git st
# On branch master
# Your branch is behind 'origin/master' by 9 commits, and can be fast-forwarded.
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       libstdc++-v3/53270.txt
#       libstdc++-v3/TODO

I've tried git read-tree -m -u HEAD but it doesn't do anything.

So my questions are:

• Why do the files reappear?
• How do I make them disappear again?
• How do I prevent them coming back?

• Is this possibly related to the fact my .git/info/exclude file contains references to files in the directories supposed to be excluded (i.e. named with !) in the sparse-checkout file? I followed the instructions to ignore the same files that SVN does

  $ git svn show-ignore >> .git/info/exclude

So my exclude files includes paths such as

# /gcc/fortran/
/gcc/fortran/TAGS
/gcc/fortran/TAGS.sub
/gcc/fortran/gfortran.info*

Which would be below one of the directories named in the sparse-checkout file:

!gcc/fortran/

I've tried to reproduce the problem with a test repo that I clone a few copies of and edit each of them, create/switch/delete branches and merge changes between them, but it never goes wrong in my toy testcases. The GCC repo is a bit big (over 2GB) and the time between "failures" (on the order of a week or two) too long to expect people to try to reproduce the problem exactly. I haven't experimented with having the same paths in sparse-checkout and exclude, as it only occurred to me today there might be a conflict there.

I asked about this on #git on freenode a few weeks ago and IIRC was basically told "it's probably a bug, noone uses sparse checkout" but I'm hoping for a better answer ;-)

Update:

The most recent time I saw the problem actually happen (i.e. the files weren't there, then appeared after a single command) was doing a pull from the upstream origin:

   bac6f1f..6c760a6  master     -> origin/master

and among the changes shown were these renames:

 create mode 100644 libgo/go/crypto/x509/root.go
 rename libgo/go/crypto/{tls => x509}/root_darwin.go (90%)
 rename libgo/go/crypto/{tls => x509}/root_stub.go (51%)
 rename libgo/go/crypto/{tls => x509}/root_unix.go (76%)
 create mode 100644 libgo/go/crypto/x509/root_windows.go

Before the pull the libgo directory was absent, as desired. After the pull that dir was present and these files (and no others) were under it:

$ ls libgo/go/crypto/x509/root_<TAB>
root_darwin.go  root_stub.go    root_unix.go    

I don't know if the renamed files lost their skip-worktree bit, how do I check that?

I'm pretty sure the problem doesn't always happen when there are renames, because e.g. the libgfortran/ChangeLog file shown in an example above is not a new file or recently renamed.

Answer 1

The skip-worktree bit can be modified with git update-index --skip-worktree. When you notice the files present you can check git ls-files -v |grep ^S (S being a file marked with skip-worktree).

But as the #git folks say, if you see odd behavior it is most likely a bug in git. After all, this is quite esoteric feature. You should probably report your findings to the git mailing list.

Edit: Also, if you are using git 1.7.7.6, I strongly recommend upgrading. 1.7.10 tree is way ahead, and I think there is a strong chance it will fix your problems.

Answer 2

In my case, I was performing some unit tests on a repo using a sparse checkout. One of my test cases created commits that contained files that were not included in my sparse checkout sub-tree list.

When I attempted to git reset --hard 123456, I received the following error:

error: Entry 'a.c' not uptodate. Cannot update sparse checkout.
fatal: Could not reset index file to revision '123456'.

The solution was to get remove the files in my working tree by re-applying the sparse-checkout rules:

git read-tree -mu HEAD

Answer 3

Check if the issue persist in the latest Git 2.13 (Q2 2017, 5 years later).
Any skip-worktree file should not be modified or even looked at during a sparse checkout anymore, because:

The preload-index code has been taught not to bother with the index entries that are paths that are not checked out by "sparse checkout".

See commit e596acc (10 Feb 2017) by Jeff Hostetler (jeffhostetler).
^{(Merged by Junio C Hamano -- gitster -- in commit c7e234f, 27 Feb 2017)}

preload-index: avoid lstat for skip-worktree items

Teach preload-index to avoid lstat() calls for index-entries with skip-worktree bit set.
This is a performance optimization.

During a sparse-checkout, the skip-worktree bit is set on items that were not populated and therefore are not present in the worktree.
The per-thread preload-index loop performs a series of tests on each index-entry as it attempts to compare the worktree version with the index and mark them up-to-date.
This patch short-cuts that work.

On a Windows 10 system with a very large repo (450MB index) and various levels of sparseness, performance was improved in the {preloadindex=true, fscache=false} case by 80% and in the {preloadindex=true, fscache=true} case by 20% for various commands.