Error: KeyUsage does not allow digital signatures - Java-applet + mutual SSL

Question

We have developed a webbased Java application running in Tomcat under IIS on Windows 2008. The website has 2-way (mutual) SSL enabled in IIS requiring the client to authenticate using a x.509 certificate (PKI) as part of SSL and this works fine with all our certificates using IE.

The website also has a java-applet called ViewOne ImageViewer. This works fine with 2-way SSL with some of our certificates but with others we get the exception on the client (java 1.6) during SSL-handshake after the user has selected its authentication certificate:

security: KeyUsage does not allow digital signatures

The most obvious difference between the certificates are that EKU (Extended Key Usage) are not present on the certificates failing. The certificate working has the EKU "Client Authentication - 1.3.6.1.5.5.7.3.2".

Does anyone know if the EKU 1.3.6.1.5.5.7.3.2 is needed to run a java applet or if this can be set somewhere? Or could the error be because of something else?

Thanks!

Answer 1

The standard normally used to validate a certificate are in RFC 5280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile. Certificates can have (at least) two extensions about their usage: Key Usage and Extended Key Usage.

• Key Usage

The Key Usage extension doesn't talk specifically about client-certificates. However, if this extension is present, the digitalSignature flag must be set, since during the SSL/TLS handshake, the CertificateVerify TLS message is signed with the private key for this certificate. This is required according to this section of RFC 5280:

The digitalSignature bit is asserted when the subject public key is used for verifying digital signatures, other than signatures on certificates (bit 5) and CRLs (bit 6), such as those used in an entity authentication service, a data origin authentication service, and/or an integrity service.

(Most cipher suites will require keyAgreement too.)

• Extended Key Usage

This one if more specific about client-certificates (if the extension is present, which is recommended but not always the case):

   id-kp-clientAuth             OBJECT IDENTIFIER ::= { id-kp 2 }
   -- TLS WWW client authentication
   -- Key usage bits that may be consistent: digitalSignature
   -- and/or keyAgreement

You can find more details about this in this NSS technical note (this should apply across other products).

When you get "security: KeyUsage does not allow digital signatures", it seems to indicate that the (non-extended) Key Usage is present in the certificate you're trying to use as a client-certificate, but that digitalSignature isn't enabled. (That's something that the CA the issued these certificates should have done.)

This is not related to the applet. However, it's possible that the URL of the applet itself is protected with client-certificate authentication, which would fail because of these extensions.

One the server side, since you're running this behind IIS, it's IIS that handles the TLS/SSL certificate verification. Apache Tomcat shouldn't really care about where it got the certificate from. (In Java, you'd be able to tweak the way you verify the certificate by configuring custom TrustManagers, but that would only apply if Java (JSSE) was handling the SSL/TLS connection itself; it doesn't apply when Tomcat is behind IIS, Apache Httpd or even when it uses APR.) I'm not sure how to configure this with IIS, but there is an option in netsh http add sslcert called [ usagecheck= ] enable | disable, which sounds like it could help. It might be too lenient, though. (Use with caution.)

This being said, it seems that you get the error on the client side, before the certificate is even sent. I must admit I haven't tried, but you might be able to use a specific KeyManager that would force the use of that certificate. I'm not entirely sure whether this would work.

Just as a side note, signing applet is a different matter. To sign an applet, the certificate needs to have the Extended Key Usage for anyExtendedKeyUsage or for id-kp-codeSigning. (Signing will work otherwise, but running the applet won't.) You can find more information here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5056088