Masstransit Extremely Slow Publishing

Question

We are using Masstransit 8.0.2 with RabbitMQ(3.8.1 Erlang 22.1.5) and.Net6. The message is being published from a TCPClient application. The message publishing time is increasing gradually and taking upto 30 minutes in publishing a single message . All the messages are being published in a background TCP receiver service. The data rate is fast as 60-70 messages are received at TCP client and being published to Rabbit using Masstransit asynchornously.

• DataMessage consumer is deployed on 3 different servers with total 5 consumer having consumer utilization of 100% (RabbitMQ Portal).
• All consumers have prefetch count of 100(we have tried with 1,5,16,50,100 and prefetch count having same results)
• The message published and acknowledging rate on this queue is on avg 10/s. (RabbitMQ server configuration 4 core cpu, 16 gb ram,Disk I/O 6400)

We have tried multiple configurations by playing with prefetch count and increasing and decreasing the consumers. We have also viewed the server cpu/memory.network utilization which all are under 50% on average Bus is being starting in application Startup.

 public static void AddServiceBus(this IServiceCollection services, IConfiguration configuration, int prefetchCount = 0, params Type[] consumers)
        {
            services.AddMassTransit(x =>
            {
                if (consumers != null && consumers.Any())
                {
                    x.AddConsumers(consumers);
                }
                x.UsingRabbitMq((context, configurator) =>
                {
                    var rabbitMqSettings = configuration.GetSection(nameof(RabbitMqConfiguration)).Get<RabbitMqConfiguration>();
                    configurator.Host(rabbitMqSettings.Host, d =>
                    {
                        d.Username(rabbitMqSettings.Username);
                        d.Password(rabbitMqSettings.Password);
                    });
                    configurator.ConfigureEndpoints(context);
                    configurator.UseRetry(b =>
                    {
                        b.Immediate(3);
                    });

                    if (prefetchCount > 0)
                        configurator.PrefetchCount = prefetchCount;
                });
                x.Configure<MassTransitHostOptions>(options =>
                {
                    options.WaitUntilStarted = true;
                    options.StartTimeout = TimeSpan.FromSeconds(30);
                    options.StopTimeout = TimeSpan.FromMinutes(1);

                });
            });
        }

private async Task<ErrorCode> PublishDataAsync(BaseData Data, string _messageGuid)
        {
            try
            {
                using var scope = _serviceProviderFactory.CreateScope();
                var publishEndpoint = scope.ServiceProvider.GetRequiredService<IPublishEndpoint>();

                var DataMessage = new DataMessage(Data);
                await _publishEndpoint.Publish(DataMessage);
                _logger.LogInformation("{messgeguid} Data Published to MassTransit", _messageGuid);
                return ErrorCodes.SUCCESS;
            }
            catch (Exception e)
            {
                _logger.LogError(e, $"Message could not be published. {JsonConvert.SerializeObject(Data)}");
            }

        }

We are trying to increasing our message publishing rate. Our publishing message size is on avg 2kb.

Answer 1

There is a benchmark built into MassTransit for verifying the throughput of your message broker.

MassTransit is easily capable of over 10,000 messages a second on a decent RabbitMQ broker instance, even 5,000 using CloudAMQP from a desktop and home inte.net connection.

If you're experiencing latency issues, they're related to your.network or broker, and not a limitation of MassTransit.

Your MassTransit version is also significantly out of date, but that won't affect the performance.

Answer 2

We have upgraded the server and it solved the issue of slow publish rate. Until anything further comes this is the resolution for us.

Any pointers to find relevance to RabbitMq performance with respect to available queues.