ServiceStack.Redis：PooledRedisClientManager创建了太多连接

Question

I think I'm doing something wrong here. Before I start, a little bit of context.

Our company works with a tool called GeneXus: It's one of those code-generator tools, which has been used for years and years. It generates C# code so we can build our own assemblies and make it work with that tool. Our application deals a lot with SOAP calls, and it also makes some good use of Redis. In fact, Redis is a main piece of the whole code infrastructure.

To make it work with Genexus, we had to create a wrapper class around the ServiceStack.Redis library, so it can be used within our GeneXus code. That's how we use it inside GeneXus:

//First we check if Redis is working at all. It just pings the Redis server.
If &RedisClient.Check()

   //Here we make several calls to get and set some data. Like that:

   If &RedisClient.Exists("Some_Key")

       &MyData = &RedisClient.Get("Some_Key")

   Else      

       &MyData = FetchFromSQLServerDatabase()        
       &RedisClient.Set("Some_Key", &MyData)

   EndIf 

   //We are done with Redis, close it.

   &RedisClient.Close()

EndIf 

That was a simple example, but our wrapper is consistently used like this: Check to see if it's online, do several things and then close the client.

The call to .Close() calls the .Dispose() method under the hood.

And this is how we manage the client creation in the wrapper.

First, we have a RedisProvider class which is a singleton. Doing some tests, we ensured that the pool is created only once. We create a pool instance like this, inside the singleton RedisProvider:

Pool = new PooledRedisClientManager(
    poolSize: poolSize,
    poolTimeOutSeconds: timeout,
    readWriteHosts: hosts);

and this RedisProvider class also has a method like this:

public RedisClient GetClient() => (RedisClient)Pool.GetClient();

What we discovered so far:

We did some tests using Apache JMeter against our SOAP webservice, simulating 50 users or so. This is what we discovered so far:

• The problem only happens inside an IIS ASP.NET application. Testing it on a Console Application with heavy concurrency fails to reproduce the problem.
• The pool itself is being created only once. The whole application shares this single instance.
• In the GeneXus example above, it's completely proven that a single connection is used across the calls, from &RedisClient.Check() to &RedisClient.Close() .
• BUT when another &RedisClient.Check() is called, normally another connection is created (and apparently it doesn't reuse the previously closed client) and we end up having thousands and thousands (assuming a pool limit of 5000) of TCP connections (which is kinda huge) in a Close Wait state, which don't get reused.
• When it hits the pool limit, we have some handling logic (which I didn't put here) to just create a new connection using new RedisClient() after the pool timeouts, which one might think it's not the smartest way to handle that, but well... It does that for a while, and then all those thousands of connections in a Close Wait state starts to close, and then the pool starts working again.

My question is: Why isn't it reusing the TCP connections? It works fine in a Console Application simulation, but when we put it to work on our Genexus application using IIS, it just keeps creating those connections.

Did I just got this pool thing wrong all the time, or am I doing something wrong?

Note: For now I'm providing all these info, but if you need more, no problem. I just don't know what more to provide.

Edit: Solved. My code was trying to be too smart. I dumbed it down and now it's working properly, though I still don't understand what I was doing wrong. Also, my assumption that absolutely all the connections to Redis were being closed right after being used turned out to be wrong.

Answer 1

The typical usage pattern for accessing a client is to use a using statement, ie:

using (var redis = redisManager.GetClient())
{
    //...
}

Calling Dispose() is what releases the client back into the pool.

Connection Pool Stats

You can view a snapshot of internal stats of the connection pool by printing out the Dictionary returned by GetStats() :

redisManager.GetStats().PrintDump();

Redis Stats

You can also view an overall stats of all Redis Client activity with the Global:

RedisStats.ToDictionary().PrintDump();

I'd also consider reducing your connection pool size as a connection pool of 5000 is close to not having a connection pool at. I'd be aiming for ~2-3x of your active connections.