Azure Event Hub - consuming messages one by one

Question

Is it the desired state that I have eg 300 outgoing messages (taken from azure event hub metrics) even though I just handled only one event? I am curious how to consume events only when the previous one was fully handled and checkpointed stored.

Otherwise, if more than one message is consumed by the consumer and the checkpoint is set after the on_event callback is finished (on_event callback takes 10 s in my case), then adding another consumer to the same consumer group and rebalancing partition ownership cause consuming many duplicates.

I am aware of that window where duplications might happen but I would like to have them as few as possible.

FYI I use python sync consumer.

Answer 1

If you really want to consume messages one by one then Event Hub is not the right solution. You should use Azure Service Bus Queues

Event Hub is not designed for one by one message processing scenario's. Instead it is build for low latency / high throughput. You might be able to come close by having just one partition (and just one consumer in just one consumer group) but you will alway have to risk of processing an already processed event in case the checkpointing fails or something like that.

If you worry about duplication the be aware that most messaging solutions have an at-least-once delivery model of messages ( source ). So make sure you processing logic is idempotent.