Why does TCP socket slow down if done in multiple system calls?

Question

Why is the following code slow? And by slow I mean 100x-1000x slow. It just repeatedly performs read/write directly on a TCP socket. The curious part is that it remains slow onl

Answer 1

Interesting. You are being a victim of the Nagle's algorithm together with TCP delayed acknowledgements.

The Nagle's algorithm is a mechanism used in TCP to defer transmission of small segments until enough data has been accumulated that makes it worth building and sending a segment over the network. From the wikipedia article:

Nagle's algorithm works by combining a number of small outgoing messages, and sending them all at once. Specifically, as long as there is a sent packet for which the sender has received no acknowledgment, the sender should keep buffering its output until it has a full packet's worth of output, so that output can be sent all at once.

However, TCP typically employs something known as TCP delayed acknowledgements, which is a technique that consists of accumulating together a batch of ACK replies (because TCP uses cumulative ACKS), to reduce network traffic.

That wikipedia article further mentions this:

With both algorithms enabled, applications that do two successive writes to a TCP connection, followed by a read that will not be fulfilled until after the data from the second write has reached the destination, experience a constant delay of up to 500 milliseconds, the "ACK delay".

(Emphasis mine)

In your specific case, since the server doesn't send more data before reading the reply, the client is causing the delay: if the client writes twice, the second write will be delayed.

If Nagle's algorithm is being used by the sending party, data will be queued by the sender until an ACK is received. If the sender does not send enough data to fill the maximum segment size (for example, if it performs two small writes followed by a blocking read) then the transfer will pause up to the ACK delay timeout.

So, when the client makes 2 write calls, this is what happens:

1. Client issues the first write.
2. The server receives some data. It doesn't acknowledge it in the hope that more data will arrive (so it can batch up a bunch of ACKs in one single ACK).
3. Client issues the second write. The previous write has not been acknowledged, so Nagle's algorithm defers transmission until more data arrives (until enough data has been collected to make a segment) or the previous write is ACKed.
4. Server is tired of waiting and after 500 ms acknowledges the segment.
5. Client finally completes the 2nd write.

With 1 write, this is what happens:

1. Client issues the first write.
2. The server receives some data. It doesn't acknowledge it in the hope that more data will arrive (so it can batch up a bunch of ACKs in one single ACK).
3. The server writes to the socket. An ACK is part of the TCP header, so if you're writing, you might as well acknowledge the previous segment at no extra cost. Do it.
4. Meanwhile, the client wrote once, so it was already waiting on the next read - there was no 2nd write waiting for the server's ACK.

If you want to keep writing twice on the client side, you need to disable the Nagle's algorithm. This is the solution proposed by the algorithm author himself:

The user-level solution is to avoid write-write-read sequences on sockets. write-read-write-read is fine. write-write-write is fine. But write-write-read is a killer. So, if you can, buffer up your little writes to TCP and send them all at once. Using the standard UNIX I/O package and flushing write before each read usually works.

(See the citation on Wikipedia)

As mentioned by David Schwartz in the comments, this may not be the greatest idea for various reasons, but it illustrates the point and shows that this is indeed causing the delay.

To disable it, you need to set the TCP_NODELAY option on the sockets with setsockopt(2).

This can be done in tcpConnectTo() for the client:

int tcpConnectTo(const char* server, const char* port)
{
    struct sockaddr_in sa;
    if(getsockaddr(server,port,(struct sockaddr*)&sa)<0) return -1;
    int sock=tcpConnect(&sa); if(sock<0) return -1;

    int val = 1;
    if (setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &val, sizeof(val)) < 0)
        perror("setsockopt(2) error");

    return sock;
}

And in tcpAccept() for the server:

int tcpAccept(const char* port)
{
    int listenSock, sock;
    listenSock = tcpListenAny(port);
    if((sock=accept(listenSock,0,0))<0) return fprintf(stderr,"Accept failed\n"),-1;
    close(listenSock);

    int val = 1;
    if (setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &val, sizeof(val)) < 0)
        perror("setsockopt(2) error");

    return sock;
}

It's interesting to see the huge difference this makes.

If you'd rather not mess with the socket options, it's enough to ensure that the client writes once - and only once - before the next read. You can still have the server read twice:

for(i=0;i<4000;++i)
{
    if(amServer)
    { writeLoop(sock,buf,10);
        //readLoop(sock,buf,20);
        readLoop(sock,buf,10);
        readLoop(sock,buf,10);
    }else
    { readLoop(sock,buf,10);
        writeLoop(sock,buf,20);
        //writeLoop(sock,buf,10);
        //writeLoop(sock,buf,10);
    }
}