How to force Apache 2.2 to send the full certificate chain?

Question

We are using Apache 2.2.25 with mod_ssl in the reverse proxy mode using mod_proxy. It has a server certificate we use for testing purposes, issued by GoDaddy. There are 3 certificates in the chain, server cert -> GoDaddy intermediate CA -> GoDaddy Root CA. The intermediate CA (Go Daddy Secure Certificate Authority - G2) is not always found in clients' list of trusted CA.

The SSL connection to the server works well for browsers (at least for some), but not for some other clients. We noticed that our server does not send the full certificate chain, by using the following command: openssl s_client -showcerts -connect SERVER_URL:443, and indeed the command reports the error Verify return code: 21 (unable to verify the first certificate)

We use the SSLCertificateFile directive in each VirtualHost:

SSLCertificateFile certificate.crt

Where the certificate.crt file contains the private key and all the certificates in the chain. We tried to split it into the following:

SSLCertificateFile server.crt
SSLCertificateKeyFile server.key
SSLCertificateChainFile chain.crt

But this didn't change anything.

Thanks for your help!

EDIT
The plot thickens - it seems to be some combination of the certificate and the server.
(testing is done with the SSL Shopper tool)

1. Go Daddy certificate (as above) on Apache 2.2 (RHEL) - does not work
2. same certificate, on IIS7 - works
3. customer's certificate (from Comodo) on Apache 2.2 RHEL - works

Answer 1

You are on the right track.

SSLCertificateFile server.crt      >> Your public certificate
SSLCertificateKeyFile server.key   >> Your private key
SSLCertificateChainFile chain.crt  >> List of intermediate certificates;
                                 in your case, only one - GoDaddy intermediate CA

Check your server configuration with a tool like SSL Labs to determine if you are sending the correct intermediate certificate.

Answer 2

You can also use the SSLCACertificatePath directive and put the original .crt files into the directory specified. However, you also have to create hash symlinks to them. This is done with the c_rehash tool, which is part of openssl. For example,

sudo c_rehash /etc/apache2/ssl/certs

However, note that there are two hash algorithms in use. The new one was introduced with openssl 1.0 and it's necessary to re-run c_rehash after upgrading openssl to 1.0 or later. This will create both old-style and new-style symlinks.

If you don't do this, openssl (and therefore apache) won't be able to find the intermediate certificates and so they won't be sent to the client. I spent a frustrating few hours debugging SSL errors after upgrading an Ubuntu server from Lucid to Precise, which had included an upgrade of openssl from 0.9.8 to 1.0.1. I searched but couldn't find any clues on the web about what was going wrong, so had to figure it out myself.

For the record, we weren't getting errors in the browser because it has a bigger set of roots and one of our intermediate certificates must have been in that set. The problem only showed up when using openssl-based command-line programs such as wget, curl and openssl s_client.