How do Jetty and other containers leverage NIO while sticking to the Servlet specification?

Question

I\'m new to NIO, and I am trying to figure out how Jetty leverages NIO.

My understanding of how traditional servlet containers that use Blocking IO service a re

Answer 1

You don't have it exactly correct. When jetty uses an NIO connector (and 9 only supports NIO) it works as follows:

1. Idle state as a few threads (1-4 depending on # cores) calling the selector, looking for IO activity. This has been scaled to over 1,000,000 connections on Jetty.

2. When selector sees IO activity, it calls a handle method on the connection, which either:

   • something else has registered that it is blocked waiting for IO for this connection, so in that case the selector just wakes up whoever was blocked.
   • otherwise a thread is dispatched to handle the connection.

3. If a thread is dispatched, then it will attempt to read the connection and parse it. What happens now depends on if the connection is http, spdy, http2 or websocket.

   • for http, if the request headers are complete, the thread goes on to call the handling of the request (eventually this gets to the servlet) without waiting for any content.
   • for http2/spdy another dispatch is required, but see the discussion about Eat-What-You-Kill strategy on the list: http://dev.eclipse.org/mhonarc/lists/jetty-dev/msg02166.html
   • for websocket the message handling is called.

4. Once a thread is dispatched to a servlet, it looks to it like the servlet IO is blocking, but underneath the level of HttpInputStream and HttpOutputStream all the IO is async with callbacks. The blocking API uses a special blocking callback to achieve blocking. This means that if the servlet chooses to use async IO, then it is just bypassing the blocking callback and using the async API more or less directly.

5. A servlet can suspend using request.startAsync, in which case the dispatched thread is returned to the thread pool, but the associated connection is not marked as interested in IO. Async IO can be performed, but a AsyncContext event is need to either reallocate a thread or to re-enroll the connection for IO activity once the async cycle is complete.

This view is slightly complicated by http2 and spdy, which are multiplexed, so they can involve an extra dispatch.

Any HTTP framework that does not dispatch can go really really fast in benchmark code, but when faced with a real application that can do silly things like block on databases, files system, REST services etc... then lack of dispatch just means that one connection can hold up all the other connections on the system.

For some more info on how jetty handles async and dispatch see:

• https://webtide.com/asynchronous-callbacks/
• https://webtide.com/jetty-9-goes-fast-with-mechanical-sympathy/
• https://webtide.com/avoiding-parallel-slowdown-in-jetty-9/
• https://webtide.com/lies-damned-lies-and-benchmarks-2/
• https://webtide.com/jetty-in-techempower-benchmarks/