我们应该使用什么gRPC监控和grpc-java指标？

Question

We use many grpc channels in a production environment. Some of the channels are opened then shutdown, and many are left open persistently.

Recently after a development project, we realized we were leaving some channels open instead of shutting them down, and we didn't realize it until it became a real hassle to fix.

We would like to get some monitoring in place for the connections.

I have found https://github.com/grpc/grpc-java/blob/master/documentation/monitoring-service-tutorial.md but it says

Note: The monitoring service requires the instrumentation-java library implementations, which are still being developed. The steps in this tutorial will not work until the instrumentation-java implementation is released.

I am looking for some simple monitoring for our grpc. Such as channels open, throughput, error counts the basic stuff.

Just wondering what your teams are using for monitoring grpc java in production?

Answer 1

You can get metrics using interceptors.

java-grpc-prometheus provides server and client side metrics (channels, message by method, etc)

Besides that, you can instrument your code with Opentracing spans to make easier to follow and measure whole transactions.

Answer 2

I've a GitHub project that monitors the following for each RPC call:

1. Service name (as in proto file)
2. Method name (as in proto file)
3. Timing (latency)
4. Status
5. Site (CLIENT/SERVER)

It also monitors the server and client side thread pool information.

https://github.com/asarkar/kotlin/tree/master/grpc-learning