What are the various thread groups in grpc

Question

I have taken the thread dump of the java application to investigate the latency issue. I see there are a lot of GRPC threads in the timed_waiting state.

I am new to grpc and want to understand the following things.

1. What is the function of grpc-default-worker threads?
2. What is the function of grpc-default-executor threads?
3. IO threads are waiting on blockingStub calls. So, On which thread/resource is it waiting on?

grpc-default-worker-ELG-3-1
THREAD ID: 330

STATE: RUNNABLE

stackTrace:
java.lang.Thread.State: RUNNABLE
at app//io.grpc.netty.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
at app//io.grpc.netty.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:132)
at app//io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:286)
at app//io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
at app//io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at app//io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at app//io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)
Locked ownable synchronizers:
- None

grpc-default-executor-76
THREAD ID : 1768

STATE : TIMED_WAITING


stackTrace:
java.lang.Thread.State: TIMED_WAITING
at java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for <6af9e3ce> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11.0.11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at java.base@11.0.11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at java.base@11.0.11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)
Locked ownable synchronizers:
- None

io-pool-145
THREAD ID : 1683

STATE : WAITING


stackTrace:
java.lang.Thread.State: WAITING
at java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for (a io.grpc.stub.ClientCalls$ThreadlessExecutor)
at java.base@11.0.11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at app//io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:731)
at app//io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:149)
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx
at java.base@11.0.11/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)
Locked ownable synchronizers:
- Locked <41493122> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Answer 1

grpc-default-worker-ELG

Responsible of the IO: doing socket read() and write() calls .

This is often called the .network thread" .

ELG is the Netty server's eventloop that gRPC chooses as default.

---

grpc-default-executor

This is the default channel executor used for running connection management tasks and executing the callbacks of the RPC.

When a new message arrives from the.network, it is read on the eventloop , and then propagated to the executor. The executor takes the messages and passes them to your ServerCall.Listener which will actually process the data.

---

IO threads are waiting on blockingStub calls. So, On which thread/resource is it waiting on?

On the worker side, as the client is waiting for new messages to come from the socket.