简体繁体 English

Azure 事件中心 - 一条一条地消费消息

[英]Azure Event Hub - consuming messages one by one

原文 2021-05-19 11:00:35 8 1 python/ azure/ azure-eventhub/ consumer

Is it the desired state that I have eg 300 outgoing messages (taken from azure event hub metrics) even though I just handled only one event?即使我只处理了一个事件，我有例如 300 条传出消息（取自 azure 事件中心指标）是否是所需的 state？ I am curious how to consume events only when the previous one was fully handled and checkpointed stored.我很好奇如何仅在前一个事件被完全处理和检查点存储时才使用事件。

Otherwise, if more than one message is consumed by the consumer and the checkpoint is set after the on_event callback is finished (on_event callback takes 10 s in my case), then adding another consumer to the same consumer group and rebalancing partition ownership cause consuming many duplicates.否则，如果消费者消费了多个消息，并且在 on_event 回调完成后设置了检查点（在我的情况下，on_event 回调需要 10 秒），那么将另一个消费者添加到同一个消费者组并重新平衡分区所有权会导致消耗很多重复。

I am aware of that window where duplications might happen but I would like to have them as few as possible.我知道 window 可能会发生重复，但我希望它们尽可能少。

FYI I use python sync consumer.仅供参考，我使用 python 同步消费者。

1 个解决方案

If you really want to consume messages one by one then Event Hub is not the right solution.如果您真的想一一消费消息，那么事件中心不是正确的解决方案。 You should use Azure Service Bus Queues您应该使用Azure 服务总线队列

Event Hub is not designed for one by one message processing scenario's.事件中心不是为一一消息处理方案而设计的。 Instead it is build for low latency / high throughput.相反，它是为低延迟/高吞吐量而构建的。 You might be able to come close by having just one partition (and just one consumer in just one consumer group) but you will alway have to risk of processing an already processed event in case the checkpointing fails or something like that.您可能可以通过仅拥有一个分区（并且仅在一个消费者组中只有一个消费者）来接近，但如果检查点失败或类似情况，您将始终冒着处理已处理事件的风险。

If you worry about duplication the be aware that most messaging solutions have an at-least-once delivery model of messages ( source ).如果您担心重复，请注意大多数消息传递解决方案至少一次传递 model 消息（来源）。 So make sure you processing logic is idempotent.因此，请确保您的处理逻辑是幂等的。