I've recently encountered the same problem in an open-source project that I maintain. Here is how I addressed this issue:
- Sources are always available for dowload via the repository, but a release will consist of a snapshot of the sources, plus a compiled version.
- Prior to making the compiled version available, I sign the assemblies with my private key.
So in your case, whoever is preparing the release should own the key. There is no need for the library developers to know about it at all.
If end-users want to recompile and sign with their own keys, that's fine. You can distinguish between the binaries of yours and others by comparing the public key that is present in the signed assemblies. Make the public key available and others can do the same.
Managing this process gets a bit cumbersome when the InternalsVisibleToAttribute is used to refer to strong-named assemblies. You can read about how I addressed that problem here.
