How to do a zero-downtime gRPC upgrade from plaintext to mTLS?

Question

Is there a way to configure the gRPC Java client and/or server in order to support a zero downtime rollout of mTLS from plaintext?

Use case: Say you have a cluster of gRPC java server running with plaintext with active gRPC java client connected to them. Service discovery is via DNS, so if you upgraded the server to support mTLS the new instances would come up and be published via DNS to the clients.

Option 1: Configure the clients to support both, so when the the client gets a new IP from the DNS lookup, it would work with either plaintext or mTLS, perhaps by trying mTLS first and falling back to plaintext. Can this be configured?

Option 2: Configure the server (on a single port) to do both. Then you'd rollout upgrades to the clients with mTLS enabled to replace the existing plaintext ones. Then when done, disable the server fallback to plaintext. Can this be configured?

Answer 1

Interesting use-case!

Option 1: Configure the clients to support both, so when the the client gets a new IP from the DNS lookup, it would work with either plaintext or mTLS, perhaps by trying mTLS first and falling back to plaintext. Can this be configured?

This would just be the client code (in the application) trying with mTLS first and on failure try plaintext for the channel. How do you expect this to be configured? Do you think the client would decide to use (or not) mTLS based on the IP address?

Option 2: Configure the server (on a single port) to do both. Then you'd rollout upgrades to the clients with mTLS enabled to replace the existing plaintext ones. Then when done, disable the server fallback to plaintext. Can this be configured?

The server needs to use a TLS handler or plaintext for all incoming connections. Do you want the server also to do some kind of fallback or examine the initial packets to see if there is a ClientHello or such and based on that decide to use the TLS handler or not?

For xDS configured gRPC servers one can configure it to do mTLS (or TLS) based on the client IP addresses as configured in the FilterChainMatch configuration for the server. See https://github.com/grpc/proposal/blob/master/A36-xds-for-servers.md#filterchainmatch