How can I speed up my rate of sending event to AzureEventHub?

Question

I am using this simple foreach loop to send events to an Azure EventHub, problem is it seems failry slow and only sends 1-2 events / second, so I feel I must be missing something, how can I speed this up? am I sending them async or what am I doing wrong?

 if (!apiInputPutCalendarService.TimeSlotIdsToClose.Any())
                {
                    return new BadRequestObjectResult(new
                    {
                        Status = "NOK",
                        Error = "There are no timeslots to close",
                        Result = ""
                    });
                }
                else
                {
                    foreach (String calendarTimeSlotId in apiInputPutCalendarService.TimeSlotIdsToClose)
                    {
                        EventHubCDSSoftDeleteTriggerModel scanMsg = new EventHubCDSSoftDeleteTriggerModel
                        {
                            TimeSlotId = calendarTimeSlotId
                        };

                        var scanMessageJsonString = JsonConvert.SerializeObject(scanMsg);

                        await EventHubHelper.SendEventToEventHubAsync(_eventHubClient, scanMessageJsonString);

                        log.LogInformation($"Message: {scanMessageJsonString} sent!");
                    }
                }

its such a small simple message I expected at least 100s of messages to be sent / second

What am I missing here?

Answer 1

Well, have you measured the impact of the logging and the serialization and the call to TimeSlotIdsToClose ? Also, I would start by not sending the messages one by one but by sending messages in batches:

await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName))
        {
            // Create a batch of events 
            using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();

            // Add events to the batch. An event is a represented by a collection of bytes and metadata. 
            eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event")));
            eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event")));
            eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Third event")));

            // Use the producer client to send the batch of events to the event hub
            await producerClient.SendAsync(eventBatch);
            Console.WriteLine("A batch of 3 events has been published.");
        }

( source )

Do mind that the code above does not reuse the EventHubProducerClient , which is recommended and that you shouldn't ignore te result of TryAdd as that will tell you whether the message can be added to the batch (there is a size based limit per batch).

Also, since I don't know your code:

Each of the Event Hubs client types is safe to cache and use as a singleton for the lifetime of the application, which is best practice when events are being published or read regularly.

( Source )