What kind of binaries? The binaries must have come from source at some point right?
So add the source that builds those binaries as a submodule in git. Then in your build process build those binaries first before building your source. The submodule is kept in sync to a specific version of the source you know that works. You also get the benefit of being able to debug easier since you have the source.
Unless the binaries are images etc, then just store those.
If space is the issue use bitbucket as they have unlimited space.