Java Grpc：使 dns 缓存无效

Question

I have a grpc client pointing to a url which resolves to 2 IP addresses. The problem is when one server node goes down and then gets back, it's not picked by the grpc client and all the load goes to a single node.

I tried recommendation to change networkaddress.cache.ttl propetty but it didn't help. My code (in Scala)

java.security.Security.setProperty("networkaddress.cache.ttl", "30")
System.setProperty("networkaddress.cache.ttl", "30")
val channel = NettyChannelBuilder.forAddress(host, port).nameResolverFactory(
      new DnsNameResolverProvider).usePlaintext().build
val client = MyServiceGrpc.newStub(channel)

grpc version: 1.32.1

Answer 1

Assuming that DNS returns both IPs all the time (probably shuffled), then the problem is not the DNS cache. The problem is that gRPC has a working connection and so won't choose to reconnect and won't perform DNS queries.

You should configure your server with MAX_CONNECTION_AGE to force clients to reconnect occasionally to rebalance the load. When clients are disconnected from the server they trigger a new DNS resolution, so this can also be used to find new addresses (although reconnections do not wait for the DNS resolution to complete).

In Java, MAX_CONNECTION_AGE is available via NettyServerBuilder.maxConnectionAge() :

NettyServerBuilder.forPort(yourPort)
    .maxConnectionAge(30, TimeUnit.MINUTES)
    ....

You want to use as large of age as you can accept. With a time like 30 minutes, then each client will rebalance every 30 minutes. So after 15 minutes of the server restarting that server would have ¼ of the load and after 30 minutes it would have roughly ½.

Answer 2

似乎配置负载平衡策略可以完成这项工作：

NettyChannelBuilder.forAddress(host, port).defaultLoadBalancingPolicy("round_robin").usePlaintext().build()