The issue for me was that I was trying to deploy on IIS as a web service, I installed the certificate on the server, but the user that runs IIS didn't have the correct permissions on the certificate.
How to give ASP.NET access to a private key in a certificate in the certificate store?