I have an certificate which I can read using the X509Certificate2 class like this:
X509Certificate2 certificate = new X509Certificate2(@\"certificate.pem\");
Combining the certificate and key with openssl, and feeding that into the x509certificate class works:
openssl pkcs12 -export -in certificate.pem -inkey privatekey.pem -out cert-and-key.pfx
And then using this to obtain a class with assigned private key:
X509Certificate2 certificate = new X509Certificate2("cert-and-key.pfx", "password");
And then the code in my question works.