I would like to have a git repository that consists mainly of binary files.
I need to keep track of the changed, added and removed files to the repository, but I don\'t
but I don't want for git to version the content of the files themselves.
That is exactly what Git Large File Storage (LFS) does (by GitHub, April 2015).
The project is git-lfs (see git-lfs.github.com) and a tested with server supporting it: lfs-test-server:
You can store metadata only in the git repo, and the large file elsewhere.
Large files are synced separately to a configurable Git LFS server over HTTPS, so you are not limited in where you push your Git repository.
Large files are only synced from the server when they are checked out, so your local repository doesn't carry the weight of every version of every file when it is not needed.
The meta data stored in Git is extensible for future use. It currently includes a hash of the contents of the file, and the file size so clients can display a progress bar while downloading or opt out of a large download.