Store the private key outside the source control root but referenced with a relative path. That way others can make builds using their own key. They'll be strongly signed but not official builds.
Setup an automatic build which has the full source tree and the private key. The auto-build can make official signed builds which can be released nightly or whatever. That way making official builds is automatic, so the authorized person holding the private key doesn't necessarily have to be involved (although you should have some mechanism in place for validating community contributed patches before including in an official build, but that's a separate issue really).
