StackExchange.Redis.RedisTimeoutException: Timeout awaiting response

Question

I have a Redis cluster of 6 instances, 3 master and 3 slaves. My ASP .NET Core application uses it as a cache. Sometimes I get such an error:

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=5KiB, 5504ms elapsed, timeout is 5000ms), command=GET, next: GET CRM.UsersMainService.GetUserInfoAsync.vvs1@domain.org, inst: 0, qu: 0, qs: 6, aw: False, rs: DequeueResult, ws: Idle, in: 0, in-pipe: 5831, out-pipe: 0, serverEndpoint: 5.178.85.30:7002, mgr: 9 of 10 available, clientName: f0b7b81f5ce5, PerfCounterHelperkeyHashSlot: 9236, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=10,Free=32757,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) (Most recent call last)

Answer 1

As I can see from your exception message, your minimum worker process count is too low for the traffic you have.

WORKER: (Busy=10,Free=32757,Min=2,Max=32767)

You had 10 busy worker threads when this exception happened, while you had 2 worker threads for start.

When your application runs out of available threads to complete an operation, .NET starts a new one (until maximum value, of course). And waits a bit to see if an additional worker thread is needed. If your application still needs worker threads, then .NET starts another one. Then another, then another... But this requires time. It doesn't occur in 0 ms. By looking your exception message, we can see that .NET had created 8 additional worker threads (10 - 2 = 8). While the creation process, this particular Redis operation had waited and eventually timed out.

You could use ThreadPool.SetMinThreads(Int32, Int32) method at the beginning of your application to set minimum thread count. I suggest you to start with ThreadPool.SetMinThreads(10, 10) and tweak it as you test it.

Additional Read:

https://docs.microsoft.com/en-us/dotnet/api/system.threading.threadpool.setminthreads https://stackexchange.github.io/StackExchange.Redis/Timeouts.html