Rabbit MQ Pulling Message More Than Once After Ack

Question

We are having an odd issue with a Rabbit MQ Implementation. We are using Containers to scale a processor. We currently are running 2 containers as a pilot test. It is basic console application that calls a method to Process Messages from the Queue.

The payload we are processing has a Guid. We are noticing the same Guid being pulled from the Queue multiple times. Event after an Ack is made on the message. This shouldn't be happening from our understanding of rabbit MQ. It might have to do with our current implementation we are using C# RabbitMQClient Library. In addition it could be the use of docker containers for our consumers. We haven't been able to reproduce this problem accept in prod.

This is only happening in one container for the Guid. So the thought is this isolated to some issue in the actual processor itself. If you need more full logs please ask.

Current Architecture

A couple of thoughts that or ideas

• Maybe the ack is not reaching fast enough before the next attempt to dequeue the message.
• Something is up with the our implementation that somebody on here can point out. We are trying to build a one queue multiple consumer model to process messages faster.

Questions:

1. What are the thoughts on implementation for the scenario above fo you experienced rabbit MQ 'ers out there?
   
2. What could be happening? (An Shell of the code with out the calls is below along with the logs of an example)

public class RabbitMQClient : IQueueClient
{
private IConnectionFactory _factory;
private IConnection _connection;
private ILoggerClient _logger;
private IWebApiClient _webApiClient;
private string _queueName;
private string _dlqName;
private string _rqName;
private int _maxRetryCount = 0;
private int _expiration = 0;
private decimal _expirationExponent = 0;

public RabbitMQClient(IConfigurationRoot config, ILoggerClient logger, IWebApiClient webApiClient)
{
    //Setup the ConnectionFactory
    _factory = new ConnectionFactory()
    {
        UserName = config["RabbitMQSettings:Username"],
        Password = config["RabbitMQSettings:Password"],
        VirtualHost = config["RabbitMQSettings:VirtualHost"],
        HostName = config["RabbitMQSettings:HostName"],
        Port = Convert.ToInt32(config["RabbitMQSettings:Port"]),
        AutomaticRecoveryEnabled = true,
        RequestedHeartbeat = 60,
        Ssl = new SslOption()
        {
            ServerName = config["RabbitMQSettings:HostName"],
            Version = SslProtocols.Tls12,
            CertPath = config["RabbitMQSettings:SSLCertPath"],
            CertPassphrase = config["RabbitMQSettings:SSLCertPassphrase"],
            Enabled = true
        }
    };

    _logger = logger;
    _webApiClient = webApiClient;

    _queueName = config["RabbitMQSettings:QueueName"];
    _dlqName = $"{_queueName}.dlq";
    _rqName = $"{_queueName}.rq";
    _maxRetryCount = int.Parse(config["RabbitMQSettings:MessageSettings:MaxRetryCount"]);
    _expiration = int.Parse(config["RabbitMQSettings:MessageSettings:Expiration"]);
    _expirationExponent = decimal.Parse(config["RabbitMQSettings:MessageSettings:ExpirationExponent"]);
}

public void ProcessMessages()
{
    using (_connection = _factory.CreateConnection())
    {
        using (var channel = _connection.CreateModel())
        {
            /*
             * Create the DLQ.
             * This is where messages will go after the retry limit has been hit.
             */
            channel.ExchangeDeclare(_dlqName, "direct");
            channel.QueueDeclare(_dlqName, true, false, false, null);
            channel.QueueBind(_dlqName, _dlqName, _queueName);

            /*
             * Create the main exchange/queue. we need to explicitly declare
             * the exchange so that we can push items back to it from the retry queue
             * once they're expired.
             */
            channel.ExchangeDeclare(_queueName, "direct");
            channel.QueueDeclare(_queueName, true, false, false, new Dictionary<String, Object>
            {
                { "x-dead-letter-exchange", _dlqName }
            });
            channel.QueueBind(_queueName, _queueName, _queueName);

            /*
             * Set the DLX of the retry queue to be the original queue
             * This is needed for the exponential backoff
             */
            channel.ExchangeDeclare(_rqName, "direct");
            channel.QueueDeclare(_rqName, true, false, false, new Dictionary<String, Object>
            {
                { "x-dead-letter-exchange", _queueName }
            });
            channel.QueueBind(_rqName, _rqName, _queueName);                    

            channel.BasicQos(0, 1, false);

            Subscription subscription = new Subscription(channel, _queueName, false);

            foreach (BasicDeliverEventArgs e in subscription)
            {
                Stopwatch stopWatch = new Stopwatch();
                try
                {
                    var payment = (CreditCardPaymentModel)e.Body.DeSerialize(typeof(CreditCardPaymentModel));

                    _logger.EventLog("Payment Dequeued", $"PaymentGuid:{payment.PaymentGuid}");

                    stopWatch.Start();

                    var response = //The Call to the Web API Happens here we will either get a 200 or a 400 from the WebService

                    stopWatch.Stop();

                    var elapsedTime = stopWatch.Elapsed.Seconds.ToString();

                    if (response.ResponseStatus == HttpStatusCode.BadRequest)
                    {
                        var errorMessage = $"PaymentGuid: {payment.PaymentGuid} | Elapsed Call Time: {elapsedTime} | ResponseStatus: {((int)response.ResponseStatus).ToString()}"
                                           + $"/n ErrorMessage: {response.ResponseErrorMessage}";
                        _logger.EventLog("Payment Not Processed", errorMessage);
                        Retry(e, subscription, errorMessage, payment.PaymentGuid);
                    }
                    else
                    {

                        //All the Responses are making it here. But even after the ACK they are being picked up and processoed again.
                        subscription.Ack(e);
                        _logger.EventLog("Payment Processed", $"--- Payment Processed - PaymentGuid : {payment.PaymentGuid} | Elapsed Call Time: {elapsedTime} | SourceStore : {payment.SourceStore} | Request Response: {(int)response.ResponseStatus}");
                    }
                }
                catch (Exception ex)
                {
                    Retry(e, subscription, ex.Message);
                    _logger.ErrorLog("Payment Not Processed", ex.ToString(), ErrorLogLevel.ERROR);
                }
            }
        }
    }
}

    public void Retry(BasicDeliverEventArgs payload, Subscription subscription, string errorMessage, Guid paymentGuid = new Guid())
    {

        if(paymentGuid != Guid.Empty)
        {
            _logger.EventLog("Retry Called", $"Retry on Payment Guid {paymentGuid}");
        }
        else
        {
            _logger.EventLog("Retry Called", errorMessage);
        }

        //Get or set the retryCount of the message
        IDictionary<String, object> headersDict = payload.BasicProperties.Headers ?? new Dictionary<String, object>();
        var retryCount = Convert.ToInt32(headersDict.GetValueOrDefault("x-retry-count"));

        //Check if the retryCount is still less than the max and republish the message
        if (retryCount < _maxRetryCount)
        {
            var originalExpiration = Convert.ToInt32(headersDict.GetValueOrDefault("x-expiration"));
            var newExpiration = Convert.ToInt32(originalExpiration == 0 ? _expiration : originalExpiration * _expirationExponent);

            payload.BasicProperties.Expiration = newExpiration.ToString();
            headersDict["x-expiration"] = newExpiration;
            headersDict["x-retry-count"] = ++retryCount;

            payload.BasicProperties.Headers = headersDict;

            subscription.Model.BasicPublish(_rqName, _queueName, payload.BasicProperties, payload.Body);
            subscription.Ack(payload);
        }
        else //Reject the message, which will send it to the DLX / DLQ
        {
            headersDict.Add("x-error-msg", errorMessage);
            payload.BasicProperties.Headers = headersDict;

            subscription.Nack(payload, false, false);
            _logger.ErrorLog("Error", errorMessage, ErrorLogLevel.ERROR);
        }
    }
}

public static class DictionaryExtensions
{
    public static TValue GetValueOrDefault<TKey, TValue>(this IDictionary<TKey, TValue> dic, TKey key)
    {
        return (dic != null && dic.TryGetValue(key, out TValue result)) ? result : default(TValue);
    }
}
}

These are the container logs and what we are seeing. You can see multiple pulls of the same payment guid even though it was successful.

Container 1

Main
AutomaticPaymentQueue
1
EventName: Payment Dequeued | EventMessage: PaymentGuid:32d065a9-57e8-4359-afac-b7339b4904cc
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : 32d065a9-57e8-4359-afac-b7339b4904cc | Elapsed Call Time: 9 | SourceStore : C0222 | Request Response: 200
EventName: Payment Dequeued | EventMessage: PaymentGuid:65ad87a8-4cfe-47e8-863c-88e0c83fcd6f
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : 65ad87a8-4cfe-47e8-863c-88e0c83fcd6f | Elapsed Call Time: 2 | SourceStore : C0222 | Request Response: 200
EventName: Payment Dequeued | EventMessage: PaymentGuid:5dc2d38f-cbc9-492b-bd41-37531974c66d
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : 5dc2d38f-cbc9-492b-bd41-37531974c66d | Elapsed Call Time: 2 | SourceStore : C0222 | Request Response: 200
EventName: Payment Dequeued | EventMessage: PaymentGuid:5dc2d38f-cbc9-492b-bd41-37531974c66d
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : 5dc2d38f-cbc9-492b-bd41-37531974c66d | Elapsed Call Time: 1 | SourceStore : C0222 | Request Response: 200
EventName: Payment Dequeued | EventMessage: PaymentGuid:dad2616c-924d-4255-ad91-a262e3bcd245
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : dad2616c-924d-4255-ad91-a262e3bcd245 | Elapsed Call Time: 1 | SourceStore : C0222 | Request Response: 200

Container 2

Main
AutomaticPaymentQueue
1
EventName: Payment Dequeued | EventMessage: PaymentGuid:cb4fcb7a-48a7-422f-86d4-69c881366f05
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : cb4fcb7a-48a7-422f-86d4-69c881366f05 | Elapsed Call Time: 4 | SourceStore : C0222 | Request Response: 200
EventName: Payment Dequeued | EventMessage: PaymentGuid:5dc2d38f-cbc9-492b-bd41-37531974c66d
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : 5dc2d38f-cbc9-492b-bd41-37531974c66d | Elapsed Call Time: 2 | SourceStore : C0222 | Request Response: 200
EventName: Payment Dequeued | EventMessage: PaymentGuid:dad2616c-924d-4255-ad91-a262e3bcd245
EventName: Payment Processed | EventMessage: --- Payment Processed - PaymentGuid : dad2616c-924d-4255-ad91-a262e3bcd245 | Elapsed Call Time: 2 | SourceStore : C0222 | Request Response: 200

The Class Publishing the Messages

public class RabbitMQClient : IQueueClient
{
    private static ConnectionFactory _factory;
    private static IConnection _connection;
    private static IModel _model;
    private const string QueueName = "AutomaticPaymentQueue";



    private void CreateConnection()
    {
        _factory = new ConnectionFactory();

        //Basic Login Infomration
        _factory.UserName = ConfigurationManager.AppSettings["RabbitMQUserName"]; ;
        _factory.Password = ConfigurationManager.AppSettings["RabbitMQPassword"];
        _factory.VirtualHost = ConfigurationManager.AppSettings["RabbitMQVirtualHost"];
        _factory.Port = Int32.Parse(ConfigurationManager.AppSettings["RabbitMQPort"]);


        //TLS Settings
        _factory.HostName = ConfigurationManager.AppSettings["RabbitMQHostName"];
        _factory.Ssl.ServerName = ConfigurationManager.AppSettings["RabbitMQHostName"];

        //SSL
        _factory.Ssl.Version = SslProtocols.Tls12;
        _factory.Ssl.CertPath = ConfigurationManager.AppSettings["RabbitMQSSLCertPath"];
        _factory.Ssl.CertPassphrase = ConfigurationManager.AppSettings["RabbitMQSSLCertPassphrase"];
        _factory.Ssl.Enabled = true;

        _connection = _factory.CreateConnection();
        _model = _connection.CreateModel();

    }

    public void SendMessage(Payload payload)
    {
         CreateConnection();
        _model.BasicPublish("", "AutomaticPaymentQueue", null, payload.Serialize());
    }
}

Answer 1

Based on the code you've provided, it looks like the problem is on the producing side. That being said, it is best practice to have message processing be an idempotent operation. In fact, a design for idempotency is a critical assumption of almost any external interface (and I would argue it is equally-important in internal interfaces).

Even if you do manage to find and resolve the problem on the publisher, you should be aware of the fact that this will not guarantee "exactly once" delivery. No such guarantee can be made. Instead, you can have one of two things (being mutually exclusive):

• At most once delivery (0 < n <= 1)
• At least once delivery (1 <= n)

From the RabbitMQ documentation :

Use of acknowledgements guarantees at-least-once delivery . Without acknowledgements, message loss is possible during publish and consume operations and only at-most-once delivery is guaranteed.

Several things are happening when messages are published and consumed. Because of the asynchronous nature of message handling systems, and the AMQP protocol in particular, there is no way to guarantee exactly once processing while still yielding the performance you would need from a messaging system (essentially, it forces everything through a serial process at the point of de-duplication). You have elected to have at least once processing, so your system needs to be designed such that a duplicate is not going to result in an unwanted change of state.