Linux Asynch IO - difference between aio.h and libaio.h

Question

I have started looking at a new paradigm I didn\'t know called asynch IO in Linux.

My goal is to use async IO targeting sockets to write high-performance efficient serve

Answer 1

For purpose of network programming, you will want event based I/O, implemented (in most basic form) with select(2) call, and on some systems with poll(), epoll(), kpoll() etc.

POSIX AIO (of which Linux AIO is an implementation), doesn't necessarily work on sockets (It did at some point in Linux 2.5 codebase, but I can't be certain it's still possible).

High-performant socket I/O on Unix is instead done in event-based manner, where you process incoming events in a loop. An event can be "socket is ready for write", "socket has new data" etc., then react on them.

Answer 2

None of these are really intended for sockets.

The POSIX AIO interface creates threads that use normal blocking IO. They work with the buffer cache and should in principle even work with sockets (though I've admittedly not tried).

The Linux kernel AIO interface does not create threads to handle requests. It works exclusively in "no buffering" mode. Beware of non-obvious behaviour such as blocking when submitting requests in some situations, which you can neither foresee nor prevent (nor know about other than your program acting "weird").

What you want is nonblocking sockets (a nonblocking socket is "kind of asynchronous") and epoll to reduce the overhead of readiness notification to a minimum, and -- if you can figure out the almost non-existing documentation -- splice and vmsplice to reduce the IO overhead. Using splice/vmsplice you can directly DMA from disk to a kernel buffer and push to the network stack from there. Or, you can directly move pages from your application's address space to kernel, and push to the network.
The downside is that the documentation is sparse (to say the least) and especially with TCP, some questions remain unaddressed, e.g. when it is safe to reclaim memory.