I am new to SSL authentication and I have a requirement to authenticate two architectural components over a trust boundary with SSL (I have control over both components). I think I will need two way SSL authentication with both the server and client having certificates.
Can the certificates be self-signed? (ie. signed by the vendor, and this won't nullify using SSL in the first place? Or do I need a third party verification service to ensure the identity of the certificate?)
How does the handshake work in terms of public and private keys for both server and client?
I will have to configure the server in IIS to use the cetificate, and send the client certificate with the request (most likely by configuration using WCF?), but are there any other tasks I will have to do to make this work?
2 way authentication with client and server certificates would work. You can use self signed certificates as well. But you would need to bypass the certificate validation on both client and server if both certificates are self signed as they are not signed by a trusted authority.
In terms of installation of the certificates it should be done as below:
Server side
ServerCert.pfx installed to LocalMachine --> Personal store
ClientCert.cer installed to Local Machine -- > Trusted People store
Client Side
ClientCert.pfx installed to Current User --> Personal Store
ServerCert.cer installed to Local Machine --> Trusted Store
You would need to ship the ServerCert.cer file to the client and the client would need to ship ClientCert.cer to you.
Now if you are using self signed certificates and your client is accessing your service then he would need to bypass the certificate validation. Sample code in C# below:
System.Net.ServicePointManager.ServerCertificateValidationCallback = (sender, cert, chain, error) => { return true; };
Hope that helps.
NOTE: From security point of view its not advisable to use self signed certificates in production environment.
I agree with EJP's answer, but since you've accepted a different one, here are further details on what the issue with self-signed certificates are going to be.
Using a self-signed certificate on your server can be done. You'll just need to configure all the potential clients to use trust this certificate. This can be done by importing the certificate (without the private key) into each machine's trusted store, or within the browser's trusted certificate repository (for example, Firefox uses its own, independent of the OS on which it's running). Self-signed server certificates here are mainly a special case of a custom CA certificate, where each client needs to know about each server it may use (with the downside that self-signed certificate can't be revoked). This is OK, but this can quickly become a problem if you have than a few machines, in practical terms.
Your main problem will be about self-signed certificates as client certificates.
Client-certificate authentication is initiated by the server, which sends a TLS Certificate Request message to the client. This message contains a list of names of Certification Authorities which it's willing to accept. Clients then use this list to select which certificate to send: they look for certificates (for which they have the private key) that have an Issuer DN matching one of the names in this CA list (they can also use certificate chains, if intermediate CA certificates are required to make the link to an Issuer DN that's in this CA list).
Most clients will simply not send any client certificate at all if they can't find a certificate that matches these conditions. This is going to be your biggest problem when trying to use self-signed client certificates.
A way around this problem is to get the server to send an empty list. This gives more freedom to the client as to which certificate to choose (it's then up to the server to choose whether or not to accept it, but that's would always be the case anyway). This is explicitly allowed by TLS 1.1 and previous versions (SSLv3/TLS 1.0) are silent on this topic, although in practice, it works fine with most SSLv3/TLS 1.0 stacks too as far as I know. (This can still lead to clunky interactions when automatic certificate selection is done by the browser, for example, since it makes it difficult to make that automatic selection correctly.)
Java's X509TrustManager can be customised using getAcceptedIssuers() to return an empty CA list in the TLS Certificate Request message. As far as I know, in C#/.Net, SslStream's RemoteCertificateValidationCallback doesn't have its equivalent. As far as I know, this list is built from the machine's trusted certificates list when using IIS/SslStream. (Note that this is independent of the certificate verification itself, it's just about the server advertising which certificates it might accept.)
In addition, if you want to use client-certificate authentication with self-signed certificates, you'll have to implement your own verification callbacks in the server to perform this authentication. A simple example would be to extract the public key from the certificate you get, compare it against a pre-defined list the server has, and then use the user name you have in that pre-defined list. You just can't trust anything a self-signed certificate says unless you have an explicit way of checking its content against something you know. Implementing a callback that just says return true; will not perform any authentication all all. Anyone could build a certificate with the same names or attributes and different keys. (It would almost be better to trust each client-cert explicitly in your server trusted certificates store, although this may lead to long CA lists in the TLS Certificate Request message.)
If you don't want to rely on a commercial CA, build you own CA. It can be a bit of work (mostly administrative), but at least you'll have a cleaner resulting system. By configuring your clients and your servers with that CA certificate, you should be able to expand to more clients and servers without changes there, and you'll also be able to benefit from the normal CA list behaviour when the client certificate is requested.
Self-signed is actually more complicated in the end. Don't do it unless you are in full control of both ends.
The private keys are used to sign the certificates being sent in the handshake so that the peer can check whether you really own the certificate you are sending. The server private key also plays a role in generating the shard secret in some ciphers.
