简体   繁体   中英

Timeouts and slow requests to Redis when using StackExchange.Redis library

We've started using Redis in our application to persist API responses there. I chose to use StackExchange.Redis to work with it and after implementing the whole logic of saving the responses the results of stress testing became quite disappointing. The total throughput of the application degraded in 3-4 times! It was about 550-600rps and now it is around 180rps (and can be even slower).

Sometimes there are no exceptions at all during a stress testing process but from time to time Timeout exceptions occur like these ones:

Timeout awaiting response (5094ms elapsed, timeout is 5000ms), inst: 0, qs: 10, in: 65536, mgr: 10 of 10 available, IOCP: (Busy=26,Free=974,Min=8,Max=1000), WORKER: (Busy=3,Free=32764,Min=8,Max=32767), v: 2.0.513.63329 
Timeout awaiting response (5063ms elapsed, timeout is 5000ms), inst: 0, qs: 4, in: 47355, mgr: 10 of 10 available, IOCP: (Busy=40,Free=960,Min=8,Max=1000), WORKER: (Busy=34,Free=32733,Min=8,Max=32767), v: 2.0.513.63329 
The timeout was reached before the message could be written to the output buffer, and it was not sent (5000ms, inst=3, qs=3, in=10, active=HMSET), inst: 3, qs: 3, in: 10, mgr: 10 of 10 available, IOCP: (Busy=36,Free=964,Min=8,Max=1000), WORKER: (Busy=34,Free=32733,Min=8,Max=32767), v: 2.0.513.63329 

And here is the total overview picture of all timeouts exceptions occurred during the last stress test: 统计

I've tried to implement connection pooling (based on TotalOutstanding property), tried to decrease the number of requests to Redis and so on but that didn't help.

I executed SHOW LOG and LATENCY DOCTOR commands but it seems that it's all okay with our Redis instance (although we've disabled THP as it was recommended by the doctor).

My assumption is that some very big API responses prevent other requests from completing. I think so because of high inbound traffic values. Is it correct? What can be done with that? Do I need to separate requests somehow?

I see an interesting hint here:

The timeout was reached before the message could be written to the output buffer

I think that, basically, Redis is telling you to clear out the incoming message because it's firehosing data at you. Likely you've created a subscription that's pretty wide in scope.

I went through the same experience, and ended up just shoving the data into a memory cache (aka ConcurrentQueue<T> ) on the subscription lambda, and processed the message results in a separate, throttled, thread, which would check the queue and process the contents, typical producer-consumer-queue approach. The timeout errors disappeared at that point.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM