问题
I am running spring cloud gateway (which I understand to be built on Spring Webflux) behind an AWS loadbalancer and I am receiving intermittent 502 errors. Upon investigation, it appears the issue has to do with connection timeouts between the loadbalancer and my nodes. From some investigation it appears that the underlying netty server has a default timeout of 10 seconds. I determined this using the following command...
time nc -vv 10.10.xx.xxx 5100
Connection to 10.10.xx.xxx 5100 port [tcp/*] succeeded!
real 0m10.009s
user 0m0.000s
sys 0m0.000s
While I could just put the idleTimeout on the load balancer to something under 10 seconds, that feels very inefficient. I would like to keep it above 30 seconds if possible. Instead I would like to increase the connection timeout on the netty server. I have attempted to set the server.connection-timeout property in my application.yml...
server:
connection-timeout: 75000
also by specifying seconds...
server:
connection-timeout: 75s
But this has had no change on the timeout when I run the time command to see how long my connection lasts, it still ends at 10 seconds...
time nc -vv 10.10.xx.xxx 5100
Connection to 10.10.xx.xxx 5100 port [tcp/*] succeeded!
real 0m10.009s
user 0m0.000s
sys 0m0.000s
What am I missing here?
回答1:
The server.connection-timeout configuration key is not supported for Netty servers (yet), I've raised spring-boot#15368 to fix that.
The connection timeout is about the maximum amount of time we should wait to for a connection to be established. If you're looking to customize the read/write timeouts, those are different options. You can add a ReadTimeoutHandler that closes the connection if the server doesn't receive data from the client in the configured duration. Same thing with a WriteTimeoutHandler, but this time about the server writing data to the client.
Here's a complete example for that:
@Configuration
public class ServerConfig {
@Bean
public WebServerFactoryCustomizer serverFactoryCustomizer() {
return new NettyTimeoutCustomizer();
}
class NettyTimeoutCustomizer implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
@Override
public void customize(NettyReactiveWebServerFactory factory) {
int connectionTimeout = //...;
int writetimeout = //...;
factory.addServerCustomizers(server -> server.tcpConfiguration(tcp ->
tcp.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectionTimeout)
.doOnConnection(connection ->
connection.addHandlerLast(new WriteTimeoutHandler(writetimeout)))));
}
}
}
Back to your question now, I've tested that configuration with the following controller:
@RestController
public class TestController {
@GetMapping(path = "/", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> textStream() {
return Flux.interval(Duration.ofSeconds(5)).map(String::valueOf);
}
}
As long as the interval is shorter than the configured write timeout, the connection is not closed by the server. You can verify that with httpie and the following command http localhost:8080/ --stream --timeout 60.
I've tested this netcat command on my local machine and I'm hitting no timeout so far.
time nc -vv 192.168.0.28 8080
192.168.0.28 8080 (http-alt) open
^CExiting.
Total received bytes: 0
Total sent bytes: 0
nc -vv 192.168.0.28 8080 0.01s user 0.00s system 0% cpu 2:36.53 total
Maybe this is something configured at the OS level, or maybe a network appliance is configured to close such connections? I just saw that you added the spring-cloud-gateway label - maybe this is something specific to that project?
回答2:
The spring documentation at https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html currently defines server.connection-timeout as "Time that connectors wait for another HTTP request before closing the connection."
This is not what that property currently does, for Netty. Right now, the property controls the TCP connection handshake timeout, which is something completely different.
There is more information about this, an example of how to actually configure an idle/keep-alive timeout at https://github.com/spring-projects/spring-boot/issues/18473
Specifically, you can use something like this:
import io.netty.channel.Channel;
import io.netty.channel.ChannelDuplexHandler;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelInitializer;
import io.netty.handler.timeout.IdleStateEvent;
import io.netty.handler.timeout.IdleStateHandler;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactory;
import org.springframework.boot.web.reactive.server.ReactiveWebServerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.time.Duration;
import static java.util.concurrent.TimeUnit.NANOSECONDS;
@Configuration
public class NettyConfig {
@Bean
public ReactiveWebServerFactory reactiveWebServerFactory(@Value("${server.netty.idle-timeout}") Duration idleTimeout) {
final NettyReactiveWebServerFactory factory = new NettyReactiveWebServerFactory();
factory.addServerCustomizers(server ->
server.tcpConfiguration(tcp ->
tcp.bootstrap(bootstrap -> bootstrap.childHandler(new ChannelInitializer<Channel>() {
@Override
protected void initChannel(Channel channel) {
channel.pipeline().addLast(
new IdleStateHandler(0, 0, idleTimeout.toNanos(), NANOSECONDS),
new ChannelDuplexHandler() {
@Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) {
if (evt instanceof IdleStateEvent) {
ctx.close();
}
}
}
);
}
}))));
return factory;
}
}
来源：https://stackoverflow.com/questions/53587611/how-to-configure-netty-connection-timeout-for-spring-webflux