CreateConnection() fails if a huge number of requests try to use IPooledObjectPolicy simultaneously for the first time

Question

I need to use RabbitMQ in my .NET application. My strategy is to have a single IConnection and multiple IModel handled by IPooledObjectPolicy .

Problem is if a huge number of requests try to use SendMessage method simultaneously for the first time, I get this error message on factory.CreateConnection(); I believe the message itself is irrelevant to the actual problem:

RabbitMQ.Client.Exceptions.BrokerUnreachableException
Message=None of the specified endpoints were reachable
IOException: connection.start was never received, likely due to a network timeout

If only one request tries to use SendMessage for the first time, everything works fine after that even if a huge number of requests triggers simultaneously.

Here is implementation of the IPooledObjectPolicy<IModel> :

private readonly RabbitMqConfigurations rabbitMqConfigurations;
private readonly IConnection _connection;

public RabbitModelPooledObjectPolicy(IOptions<RabbitMqConfigurations> RabbitMqConfigurations)
{
    rabbitMqConfigurations = RabbitMqConfigurations.Value;
    _connection = GetConnection();
}

private IConnection GetConnection()
{
    var factory = new ConnectionFactory
    {
        HostName = rabbitMqConfigurations.HostName
    };

    return factory.CreateConnection(); //<-- it fails
}

public IModel Create() => return _connection.CreateModel();

public bool Return(IModel obj)
{
    if (obj.IsOpen)
        return true;
    else
    {
        obj?.Dispose();
        return false;
    }
}

This is how I used the IPooledObjectPolicy<IModel> :

private readonly DefaultObjectPool<IModel> objectPool;

public RabbitMQProducer(IPooledObjectPolicy<IModel> objectPolicy)
{
    objectPool = new DefaultObjectPool<IModel>(objectPolicy, Environment.ProcessorCount * 2);
}

public void SendMessage<T>(T message, string queue) where T : class
{
    var channel = objectPool.Get();

    try
    {
        channel.QueueDeclare(queue: queue, durable: true, exclusive: false, autoDelete: false, arguments: null);

        var properties = channel.CreateBasicProperties();
        properties.Persistent = true;

        var json = JsonConvert.SerializeObject(message);
        var body = Encoding.UTF8.GetBytes(json);

        channel.BasicPublish(exchange: "", routingKey: queue, basicProperties: properties, body: body);
    }
    catch
    {
        throw;
    }
    finally
    {
        objectPool.Return(channel);
    }
}

I registered these two services as singleton:

services.AddSingleton<IPooledObjectPolicy<IModel>, RabbitModelPooledObjectPolicy>();
services.AddSingleton<IMessageProducer, RabbitMQProducer>();

Answer 1

I could manage to fix this.

Explanation of the problem:

RabbitModelPooledObjectPolicy constructor executes when class is accessed for the first time. While the constructor is executing, application is trying to handle other requests as well and the requests are a lot and are keep coming like a non-stop stream.

Therefore it seems somehow application is abandoning factory.CreateConnection() temporarily to handle other coming requests in order to executes factory.CreateConnection() later. However, there is no end to coming requests and the same scenario goes until factory.CreateConnection() throws a time out exception.

Solution:

I warmed up IPooledObjectPolicy<IModel> in Configure method in startup.cs to execute RabbitModelPooledObjectPolicy constructor in startup before application gets ready and receives requests.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env){
    app.ApplicationServices.GetService<IPooledObjectPolicy<IModel>>();
}