Throwing exception on NIO SSL Handshake process using SSLEngine

Question

I am getting following exception on NIO SSL handshake. During handshake process,

On client side,

a) NEED_WRAP

b) NEED_UNWRAP

c) NEED_TASK

Answer 1

You're not supposed to 'follow' a 'sequence' when using the SSLEngine. You are supposed to react to the states and exceptions it provides:

• NEED_WRAP: do a wrap() from the application send buffer to the network send buffer
• NEED_UNWRAP: do an unwrap() from the network receive buffer to the application receive buffer
• BUFFER_OVERFLOW: do a flip()/write()/compact() on the network send buffer, or a flip()/get()/compact() on the application receive buffer
• BUFFER_UNDERFLOW: do a read() on the network receive buffer, or there is nothing in the application send buffer.

EDIT What's this?

if(myNetData.limit() == 0)
{
    myNetData.clear();
}

and this?

case NEED_WRAP:
    myNetData.clear();

You can't just throw away engine data. Get rid of these altogether. You can't do anything to the network send or receive buffers except flip(), compact(), wrap(), and unwrap(). Also it doesn't appear that you have separate net send and receive buffers. You need both. You need four altogether: net send, net receive, application send, and application receive. The network buffers need to be of the sizes advised by the SSLEngine.

Apart from that, you aren't really reacting exactly as I said above. For example, take the NEED_UNWRAP path. You should:

1. unwrap() from the net recv buffer to the application receive buffer.
2. If you then get BUFFER_UNDERFLOW, read() into the net receive buffer and repeat (1).

For NEED_WRAP:

3. wrap() from the application send buffer to the network send buffer.
4. If you then get BUFFER_OVERFLOW, write() from the net send buffer and repeat 3.

When you need to read application data:

5. flip()/get()/compact() from the application receive buffer.
6. If that results in a BufferUnderflowException, unwrap() and repeat, bearing in mind that the unwrap() may cause NEED_WRAP or NEED_WRAP or BUFFER_UNDERFLOW or BUFFER_OVERFLOW.

When you need to write application data:

7. put() into the application send buffer.
8. If that results in a BufferOverflowException, flip()/wrap()/compact(), bearing in mind that the wrap() may cause NEED_WRAP or NEED_WRAP or BUFFER_UNDERFLOW or BUFFER_OVERFLOW.