Websocket SSL handshake failure

Question

I have spring-boot Tomcat server for secure websocket connections. The server accepts Android 4.4, iOS, Firefox, and Chrome clients without failure with an authority-signed

Answer 1

error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected message (external/openssl/ssl/s3_both.c:498 0xac526e61:0x00000000)
        at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake_bio(Native Method)
        at com.android.org.conscrypt.OpenSSLEngineImpl.unwrap(OpenSSLEngineImpl.java:423)

0x1408E0F4 is:

$ openssl errstr 0x1408E0F4
error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected message

It shows up in the OpenSSL sources at a couple of places:

$ cd openssl-1.0.1l
$ grep -R SSL3_GET_MESSAGE *
ssl/s3_both.c:          SSLerr(SSL_F_SSL3_GET_MESSAGE,SSL_R_UNEXPECTED_MESSAGE);
ssl/s3_both.c:          SSLerr(SSL_F_SSL3_GET_MESSAGE,SSL_R_UNEXPECTED_MESSAGE);
ssl/s3_both.c:          SSLerr(SSL_F_SSL3_GET_MESSAGE,SSL_R_EXCESSIVE_MESSAGE_SIZE);
ssl/s3_both.c:          SSLerr(SSL_F_SSL3_GET_MESSAGE,SSL_R_EXCESSIVE_MESSAGE_SIZE);
ssl/s3_both.c:          SSLerr(SSL_F_SSL3_GET_MESSAGE,ERR_R_BUF_LIB);

Here's the code I believe is causing the trouble (line numbers have changed, and the SSLerr is at 491):

/* Obtain handshake message of message type 'mt' (any if mt == -1),
 * maximum acceptable body length 'max'.
 * The first four bytes (msg_type and length) are read in state 'st1',
 * the body is read in state 'stn'.
 */
long ssl3_get_message(SSL *s, int st1, int stn, int mt, long max, int *ok)
    {
    ...

    /* s->init_num == 4 */
    if ((mt >= 0) && (*p != mt))
        {
        al=SSL_AD_UNEXPECTED_MESSAGE;
        SSLerr(SSL_F_SSL3_GET_MESSAGE,SSL_R_UNEXPECTED_MESSAGE);
        goto f_err;
        }
    ...

But I'm not sure what causes that particular problem. See this question on the OpenSSL User List at SSL_F_SSL3_GET_MESSAGE and SSL_R_UNEXPECTED_MESSAGE.

EDIT: according to the Android source for s3_both.c, that is the code that's triggering the issue.

-----

OK, looking at the file successful.pcap and unsuccessful.pcap, the good client is using TLS 1.0 while the misbehaving client is using TLS 1.2. But I don't see anything offensive that would cause the client to close the connection while processing the four messages (Server Hello, Certificate, Server Key Exchange, Server Hello Done) in the Record.

-----

Based on the ServerKeyExchange message:

The server selected the client's offering of secp521r1. You might want to use secp256. That's most interoperable right now. Also see Is the limited elliptic curve support in rhel/centos/redhat openssl robust enough?.

-----

OpenSSL 1.0.1e FIPS used by the server has suffered a few problems. See, for example:

• Binary curves broken in FIPS mode
• Crash when using TLS 1.2 caused by use of incorrect hash algorithm

If possible, you might want to upgrade it to something newer.

-----

Is there a way to debug the Android client side SSL implementation?

I think this is an easier question. Use a custom SSLSocketFactory like SSLSocketFactoryEx. It will allow you to try different protocols, cipher suites and settings. But its trial-and-error.

Otherwise, you would need to grab a copy of the OpenSSL source code used by Android 5.0 (including patches). I don't know how to get that and ensure it builds like mainline OpenSSL (effectively, you need to build s_client using Android sources with debugging information).

This might be helpful: OpenSSL on Android. From the looks of the diffs, it appears Android is using OpenSSL 1.0.0. (Some of the patches in the patch/ directory specifically call out 1.0.0b).