SSL Handshake fails after clienthello

Question

EDIT: I\'ll leave this as a good example of debugging SSL.

Final analysis: We had a networking issue in which one of our routers was misconfigured for a totally dif

Answer 1

As I can see, the client hello has one byte to much on position 00cb: the 0xFF should not be there. So the following bytes data cannot be read correctly...

00c0 - 00 14 00 11 00 08 00 06-00 03 00 ff 01 00 00 6d ...............m

Not sure, but it seems to be a bug in openssl, so a firewall or web proxy ignores the request.

Answer 2

Well, I had a similar problem. SSL handshake terminated with error right after ClientHello. It turned out the server required stronger ciphers than I had installed so I had to install the Java Cryptography Extension (JCE) (http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html).

More interestingly, we had the very same problem on a server and we had the jars for JCE there but some older version so it suffered the same problem. Replacing them with the newest version solved the issue.

BTW, do you know how to get required server ciphers? Or better compare the client and server ciphers? So one would immediately see the mismatch.

Answer 3

For folks who come across this issue, I created a checklist:

• Make sure all TLS versions are enabled in Internet Explorer (This is for testing. You can later disable the unsecure versions once you find out the root cause)

• Check the registry keys below to make sure what you set in Internet Explorer is applied at registry level. If there are working and non-working servers, mirror the working server’s settings HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols

• Collect a network trace from the client side. Check if the client and server are agreeing on a cipher suite. If they are not, make sure the client uses the cipher suites the server is trying to use. The Group Policy setting is below Computer Configuration > Administrative Templates > Network > SSL> Configuration Settings > SSL Cipher Suite Order

• If the issue sill exists, look for any network device (proxy, firewall, load balancer etc.) in between that might be blocking TLS traffic

• Check the website binding in IIS. Make sure the certificate is valid and the port is set to 443

• Make sure the port 443 is listened in the server (netstat -an -p TCP | find /I "listening"). More details: Port 80 and 443 are not listened in IIS server Change the port number to 444 and test website. If it is accessible, it means there is a software blocking or overwriting 443 port. More details

• Disable Windows Firewall (If it works, you can enable it back and set up rules accordingly)

• Look for any third-party applications such as antivirus or endpoint protection software in the server such as Symantec Endpoint Security and Symantec Data Center Security Server Agent (Based on this document, Security Server Agent uses port 443). Uninstall them (Don’t just disable. Uninstall completely. If it works, you can install them back and configure them accordingly)

• Check if there is any Microsoft software that is using port 443. Applications like SQL Server Reporting Services (SSRS) and Windows Admin Center might interfere the port 443. An example

Source: The missing Server Hello in TLS handshake (ERR_SSL_PROTOCOL_ERROR

Answer 4

This is old and already answered, but we suffered the same exact issue and the cause was different.

The key was to sniff traffic on our edge router, where we saw ICMP messages to the server (GitHub.com) asking for fragmentation. This was messing the connection, with retransmissions, duplicated ACKs and so.

The ICMP packet had a field, MTU of next hop with a weird value, 1450. The usual value is 1500.

We checked our router and one of the interfaces (an Ethernet tunnel) had this value as MTU, so the router was taking the minumum MTU of all interfaces as next hop. As soon as we removed this interface (it was unused), the SSL handshake started to work again.