JAVA 中的 GRPC 服务器以支持巨大的请求负载

Question

I have a grpc (1.13.x) server on java that isn't performing any computation or I/O intensive task. The intention is to check the number of requests this server can support per second on 80 core machine.

Server:

  ExecutorService executor = new ThreadPoolExecutor(160, Integer.MAX_VALUE,
                60L, TimeUnit.SECONDS,
                new SynchronousQueue<Runnable>(),
                new ThreadFactoryBuilder()
                        .setDaemon(true)
                        .setNameFormat("Glowroot-IT-Harness-GRPC-Executor-%d")
                        .build());
     Server server =  NettyServerBuilder.forPort(50051)
                .addService(new MyService())
                .executor(executor)
                .build()
                .start();

Service:

@Override
    public void verify(Request request, StreamObserver<Result> responseObserver) {
        Result result = Result.newBuilder()
                    .setMessage("hello")
                    .build();
        responseObserver.onNext(result);
        responseObserver.onCompleted();
    }

I am using ghz client to perform a load test. Server is able to handle 40k requests per second but RPS count is not able to exceed more than 40k even on increase in number of concurrent clients with incoming requests rate 100k. GRPC server is able to handle just 40K requests per second and it queues all other requests. CPU is underutilized (7%). About 90% of grpc threads (with prefix grpc-default-executor) were in waiting state, despite no I/O operation. More than 25k threads are in waiting state.

Stacktrace of threads in waiting:

grpc-default-executor-4605
PRIORITY :5

THREAD ID :0X00007F15A4440D80

NATIVE ID :
stackTrace:
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@15.0.1/Native Method)
- parking to wait for <0x00007f1df161ae20> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@15.0.1/LockSupport.java:252)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@15.0.1/SynchronousQueue.java:462)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@15.0.1/SynchronousQueue.java:361)
at java.util.concurrent.SynchronousQueue.poll(java.base@15.0.1/SynchronousQueue.java:937)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@15.0.1/ThreadPoolExecutor.java:1055)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@15.0.1/ThreadPoolExecutor.java:1116)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@15.0.1/ThreadPoolExecutor.java:630)
at java.lang.Thread.run(java.base@15.0.1/Thread.java:832)
Locked ownable synchronizers:
- None

How can I configure the server to support 100K+ requests?

Answer 1

Nothing in the gRPC stack seems to cause this limit. What's the average response time on the server side? It looks like you are limited by the ephemeral ports or TCP connection limit and you may want to tweak your kernel as described at here https://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 or here https://blog.box.com/ephemeral-port-exhaustion-and-web-services-at-scale