HAProxy closes long living TCP connections ignoring TCP keepalive

Question

I have configured HAProxy (1.5.4, but I tried also 1.5.14) to balance in TCP mode two server exposing AMQP protocol (WSO2 Message Broker) on 5672 port. The clients create and us

Answer 1

TCP keep alive is at the transport layer and is only used to do some traffic on the connection so intermediate systems like packet filters don't loose any states and that the end systems can notice if the connection to the other side broke (maybe because something crashed or a network cable broke).

TCP keep alive has nothing to do with the application level idle timeout which you have set explicitly to 200s:

timeout client 200000ms
timeout server 200000ms

This timeouts gets triggered if the connection is idle, that is if no data get transferred. TCP keep alive does not transport any data, the payload of these packets is empty.

Answer 2

The timeout client detects a dead client application on a responsive client OS. You can always have an application that occupies a connection but doesn't speak to you. This is bad because the number of connections isn't infinite (maxconn).

Similarly, set timeout server for the backend.

These options were for haproxy talking to application. Now, there is a completely separate check where OS talks to OS (without touching the app or haproxy):

With option clitcpka or option srvtcpka or option tcpka you allow the inactive connection to be detected and killed by the OS, even when haproxy doesn't actively check it. This primarily needs OS settings (Linux).

If no data sent for 110 seconds then immediately send the first keep-alive (KA), don't kill connection yet:

 sysctl net.ipv4.tcp_keepalive_time=110

Wait for 30 seconds after each KA, once they're enabled on this connection:

 sysctl net.ipv4.tcp_keepalive_intvl=30

Allow 3 KAs be unacknowledged, then kill the TCP connection:

 sysctl net.ipv4.tcp_keepalive_probes=3

In this situation OS kills the connection 200 seconds after packets stop coming.