Java Grpc: invalidate dns cache

北慕城南 提交于 2021-02-08 03:25:12

问题


I have a grpc client pointing to a url which resolves to 2 IP addresses. The problem is when one server node goes down and then gets back, it's not picked by the grpc client and all the load goes to a single node.

I tried recommendation to change networkaddress.cache.ttl propetty but it didn't help. My code (in Scala)

java.security.Security.setProperty("networkaddress.cache.ttl", "30")
System.setProperty("networkaddress.cache.ttl", "30")
val channel = NettyChannelBuilder.forAddress(host, port).nameResolverFactory(
      new DnsNameResolverProvider).usePlaintext().build
val client = MyServiceGrpc.newStub(channel)

grpc version: 1.32.1


回答1:


Assuming that DNS returns both IPs all the time (probably shuffled), then the problem is not the DNS cache. The problem is that gRPC has a working connection and so won't choose to reconnect and won't perform DNS queries.

You should configure your server with MAX_CONNECTION_AGE to force clients to reconnect occasionally to rebalance the load. When clients are disconnected from the server they trigger a new DNS resolution, so this can also be used to find new addresses (although reconnections do not wait for the DNS resolution to complete).

In Java, MAX_CONNECTION_AGE is available via NettyServerBuilder.maxConnectionAge():

NettyServerBuilder.forPort(yourPort)
    .maxConnectionAge(30, TimeUnit.MINUTES)
    ....

You want to use as large of age as you can accept. With a time like 30 minutes, then each client will rebalance every 30 minutes. So after 15 minutes of the server restarting that server would have ¼ of the load and after 30 minutes it would have roughly ½.




回答2:


Seems that configuring load-balancing policy does the job:

NettyChannelBuilder.forAddress(host, port).defaultLoadBalancingPolicy("round_robin").usePlaintext().build()


来源:https://stackoverflow.com/questions/64007975/java-grpc-invalidate-dns-cache

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!