client and server cannot communicate, because they do not possess a common algorithm - SSLStream

Question

There are lot of similar questions already answered and none of the answers help my current scenario.

In the windows service we have TCPL SSL stream and client connecting to the stream.

I have created a .NET Client and am able to access the server successfully with Strict TLS 1.2 using IIS Crypto.

We are trying to access the server using Lantronix xport Pro and the below line throws exception

stream.AuthenticateAsServer(serverCertificate, false, SslProtocols.Ssl3| SslProtocols.Tls | SslProtocols.Tls11 | SslProtocols.Tls12, True);

Exception happens only if strict TLS 1.2 is enabled. If we have SSL3 protocol enabled everything works fine without any issues.

using WireShark i can see client sending TLS 1.2 request with below ciphers

Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)

As per MSDN above Ciphers are supporting TLS 1.2

Am not having any issue connecting from the .net Client which we created for testing purposes with strict TLS 1.2

Creating a self signed certificate using open ssl. server is running with windows server 2012 R2.

Am not sure on what options i should try for SSLstream specific to TLS 1.2 ?

Is there any thing specific i need to for other than .net client to communicate ?

Any suggestions on tracing this issue would be great

Update on the Lantrix Pro request

Frame 33845: 112 bytes on wire (896 bits), 112 bytes captured (896 bits) on interface 0
Ethernet II, Src: Pronet_c7:bb:29 (00:20:4a:c7:bb:29), Dst: Vmware_99:95:c7 (00:50:56:99:95:c7)
Internet Protocol Version 4, Src: 10.10.110.71, Dst: 10.10.110.10
Transmission Control Protocol, Src Port: 38182, Dst Port: 28000, Seq: 1, Ack: 1, Len: 58
Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 53
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 49
            Version: TLS 1.2 (0x0303)
            Random: 8a f5 6f 9e 92 65 43 c7 45 9b 57 ff dd a4 22 45 ...
                GMT Unix Time: Nov 16, 2043 20:38:22.000000000 Central Standard Time
                Random Bytes: 92 65 43 c7 45 9b 57 ff dd a4 22 45 12 16 cb 41 ...
            Session ID Length: 0
            Cipher Suites Length: 10
            Cipher Suites (5 suites)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
                Cipher Suite: TLS_RSA_EXPORT1024_WITH_RC4_56_MD5 (0x0060)
                Cipher Suite: TLS_RSA_EXPORT1024_WITH_RC4_56_SHA (0x0064)
                Cipher Suite: TLS_RSA_EXPORT_WITH_RC4_40_MD5 (0x0003)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)

Answer 1

TL;DR: the client is broken. It looks like the vendor barely added minimal TLS 1.2 support to an old product while keeping insecure ciphers and failing to add support for the currently used SHA-256 signed certificates.

---

The client does only send a very minimal TLS 1.2 handshake. It only contains 5 ciphers (where the 3 EXPORT ciphers are critically insecure, the 3DES cipher is slightly insecure but the AES128-SHA is acceptable). And, compared to other TLS successful 1.2 handshakes it does not contain the signature_algorithm TLS extension.

This extension is new to TLS 1.2. It is used to tell the server which signature algorithms are supported. If the extension is not provided it will default to SHA1 with RSA for the ciphers offered. To cite from the RFC:

If the client does not send the signature_algorithms extension, the server MUST do the following:

• If the negotiated key exchange algorithm is one of (RSA, DHE_RSA, DH_RSA, RSA_PSK, ECDH_RSA, ECDHE_RSA), behave as if client had sent the value {sha1,rsa}.

But, the server provides a certificate with an RSA key but with SHA-256 as hash. Thus, this certificate will not match the accepted signature algorithms and the handshake fails. If the server instead allows TLS 1.1 or lower then the handshake will succeed since the signature_algorithm extension is ignored with these lower protocol versions and thus the server can send the SHA-256 signed certificate it has.

As can be seen from a capture with an openssl s_client in the provided pcaps the TLS 1.2 handshake works if the client provides the signature_algorithm extension and signals support for RSA and SHA-256.