Resolution of Server IPs in gRPC

Question

We are running a setup on production where grpc clients are talking to servers via proxy in between (image attached)

The client is written in java and server is written in go. We are using the load balancing property as round_robin in the client. Despite this, we have observed some bizarre behaviour. When our proxy servers scale in ie reduce from let's say 4 to 3, then resolver gets into action and the request load from our clients gets distributed equally to all of our proxies, but when the proxy servers scale out ie increase from 4 to 8, then the new proxy servers don't get any requests from the clients which leads to a skewed distribution of request load on our proxy servers. Is there any configuration that we can do to avoid this?

We tried setting a property named networkaddress.cache.ttl to 60 seconds in the JVM ARGS but even this didn't help.

Answer 1

You need to cycle the sticky gRPC connections using the keepalive and keepalive timeout configuration in the gRPC client.

Please have a look at this - gRPC connection cycling

Answer 2

both round_robin and pick_first perform name resolution only once. They are intended for thin, user-facing clients (android, desktop) that have relatively short life-time, so sticking to a particular (set of) backend connection(s) is not a problem then.

If your client is a server app, then you should be rather be using grpclb or the newer xDS : they automatically re-resolve available backends when needed. To enable them you need to add runtime dependency in your client to grpc-grpclb or grpc-xds respectively.
grpclb does not need any additional configuration or setup, but has limited functionality. Each client process will have its own load-balancer+resolver instance. backends are obtained via repeated DNS resolution by default.
xDS requires central envoy instance from which it obtains available backends.