Handling SSL_shutdown correctly

Question

The OpenSSL documentation on SSL_shutdown states that:

It is therefore recommended, to check the return value of SSL_shutdown()

Answer 1

openssl is a bit of a dark art.

Firstly the page you referenced has HTML-ified the return values badly. Here's what the man-page actually says:

  RETURN VALUES

   The following return values can occur:

   0   The shutdown is not yet finished. Call SSL_shutdown() for a second
       time, if a bidirectional shutdown shall be performed.  The output
       of SSL_get_error(3) may be misleading, as an erroneous
       SSL_ERROR_SYSCALL may be flagged even though no error occurred.

   1   The shutdown was successfully completed. The "close notify" alert
       was sent and the peer's "close notify" alert was received.

   -1  The shutdown was not successful because a fatal error occurred
       either at the protocol level or a connection failure occurred. It
       can also occur if action is need to continue the operation for non-
       blocking BIOs.  Call SSL_get_error(3) with the return value ret to
       find out the reason.

If you have blocking BIOs, things are relatively simple. A 0 on the first call means you need to call SSL_shutdown again if you want a full bidirectional shutdown. Basically it means that you sent a close_notify alert but haven't one back yet). A 1 would mean you previously received a close_notify alert from the other peer, and you're totally done. A -1 means an unrecoverable error. On the second call (which you only do if you got a 0 back), then a bidirectional shutdown is initiated (i.e. now wait from the other side for them to send you their "close_notify" alert). Logic dictates you can't get a 0 back again (because it's a blocking BIO and will have completed the first step). A -1 indicates an error, and a 1 indicates completion success.

If you have non-blocking BIOs, the same "possibly 0 then 1" return values apply, save for the fact you need to go through the whole SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE rigmarole as well, i.e.:

   If the underlying BIO is non-blocking, SSL_shutdown() will also return
   when the underlying BIO could not satisfy the needs of SSL_shutdown()
   to continue the handshake. In this case a call to SSL_get_error() with
   the return value of SSL_shutdown() will yield SSL_ERROR_WANT_READ or
   SSL_ERROR_WANT_WRITE. The calling process then must repeat the call
   after taking appropriate action to satisfy the needs of SSL_shutdown().
   The action depends on the underlying BIO. When using a non-blocking
   socket, nothing is to be done, but select() can be used to check for
   the required condition. When using a buffering BIO, like a BIO pair,
   data must be written into or retrieved out of the BIO before being able
   to continue.

So you have two levels of repetition. You call SSL_shutdown the 'first' time but repeat if you get SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE after going around the select() loop in the normal way, and only count the 'first' SSL_shutdown as done if you get a non SSL_ERROR_WANT_ error code (in which case it failed), or you get a 0 or 1 return. If you get a 1 return, you've done. If you get a 0 return, and you want a bidirectional shutdown, then you have to do the second call, on which again you will need to check for SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE and retry select; that should not return 1, but may return 0 or an error.

Not simple.

Couple more notes from the docs: after calling SSL_shutdown and getting a "0" back the first time, you could optionally then call SSL_read instead of SSL_shutdown (in case the peer is still sending you any data on that SSL socket), and, I guess, "hope" that they eventually send you a close message from their side, to flush the pipes.

Also if you're planning on closing the socket after shutdown completion "anyway" you could entirely skip the second call to SSL_shutdown (the "1" of the "0 then 1") and just go ahead and close the socket, the kernel should take care of discarding the "now ignored" close_notify alert that presumably they should be about to send...