[英]In C#, How can I get ALL the messages out of an Azure Service Bus Queue?
Because I need to remove duplicate messages and delay processing some messages which are "too new" (as determined by the most recent copy of a message), I want to handle the entire contents of Service Bus Queue all at once.因为我需要删除重复的消息并延迟处理一些“太新”的消息(由消息的最新副本确定),所以我想一次处理服务总线队列的全部内容。
I'm not sure how many messages I can expect, but I feel strongly optimistic that it shouldn't normally be in the hundreds, never mind that thousands which I thought could be the limit for ReceiveAsync (int maxMessageCount, TimeSpan operationTimeout)
.我不确定我可以期待多少条消息,但我非常乐观地认为它通常不应该是数百条,更不用说我认为可能是
ReceiveAsync (int maxMessageCount, TimeSpan operationTimeout)
的限制了。 However, it turns out that no matter how high that value is, I'm only able to read about 30 to 50 messages in a single invocation of然而,事实证明,无论该值有多高,我只能在一次调用中读取大约 30 到 50 条消息
private async Task<IList<MicrosoftMessage>> Receive(IQueueConfig queueConfig) =>
await _messageReceiverLookup.GetMessageReceiver(queueConfig.QueueName)
.ReceiveAsync(queueConfig.MaximumRecords, TimeSpan.FromSeconds(10));
I tried wrapping this with some additional logic, like:我尝试用一些额外的逻辑来包装它,比如:
List<MicrosoftMessage> messages = new();
List<MicrosoftMessage> newMessages = new();
do
{
newMessages = await ReceiveMessages(queueHandler, cancellationToken);
messages.AddRange(newMessages);
}
while (
newMessages.Count > 0
&& messages.Count > 0
&& messages.Count < queueHandler.QueueConfig.MaximumRecords
);
But discovered this never ends because the system will read the same message multiple times.但发现这永远不会结束,因为系统会多次读取相同的消息。
So then I tried this:所以我尝试了这个:
Dictionary<string, MicrosoftMessage> previosMessagesByToken;
Dictionary<string, MicrosoftMessage> allMessagesByToken = new();
List<MicrosoftMessage> newMessages;
do
{
previosMessagesByToken = allMessagesByToken;
newMessages = await ReceiveMessages(queueHandler, cancellationToken);
Dictionary<string, MicrosoftMessage> newMessagesByToken = newMessages.ToDictionary(x => x.SystemProperties.LockToken, x => x);
// Ensure we only collect each message once!
allMessagesByToken = allMessagesByToken.Concat(newMessagesByToken.Where(kvp => !allMessagesByToken.ContainsKey(kvp.Key)))
.ToDictionary(kvp => kvp.Key, kvp => kvp.Value);
}
while (
newMessages.Count > 0
&& allMessagesByToken.Count > previosMessagesByToken.Count
&& allMessagesByToken.Count < queueHandler.QueueConfig.MaximumRecords
);
This seems to work, but on the one hand, I have a gut feeling this should not be so complicated.这似乎可行,但一方面,我有一种直觉,这不应该那么复杂。 Also, I don't entirely trust this since I do not entirely understand why I don't get all the messages nor why I get duplicate messages, so I can't help feeling that somehow this algorithm could allow some messages to fall between the cracks, being a non-duplicate that is not included.
另外,我并不完全相信这一点,因为我不完全理解为什么我没有收到所有消息,也没有收到重复的消息,所以我不禁觉得这个算法可能会允许一些消息落在裂缝,是不包括在内的非重复项。
Is there better way I can make get all the messages?有没有更好的方法可以让我获取所有消息?
A few foundational assumptions:几个基本假设:
PeekLock
mode will have their lock expired at some point and will be delivered.PeekLock
模式下接收到的消息将在某个时间点的锁定过期并被传递。 If your objective to drain all the messages, you should either complete the messages you've received or receive in ReceiveAndDelete
mode.如果您的目标是耗尽所有消息,您应该完成已收到的消息或在
ReceiveAndDelete
模式下接收。 That way you won't get the same messages again.这样你就不会再收到相同的消息了。 If you trying to peek at the messages in the queue, then your
LockDuration
would need to be long enough to ensure all messages have peeked.如果您试图查看队列中的消息,那么您的
LockDuration
需要足够长以确保所有消息都已被查看。
I need to remove duplicate messages and delay processing some messages which are "too new" (as determined by the most recent copy of a message), I want to handle the entire contents of Service Bus Queue all at once.
我需要删除重复的消息并延迟处理一些“太新”的消息(由消息的最新副本确定),我想一次处理服务总线队列的全部内容。
The bigger issue seems to be the attempt to handle messages in the queue like records in a database.更大的问题似乎是尝试处理队列中的消息,如数据库中的记录。 Duplicate detection is already a feature of Azure Service Bus.
重复检测已经是 Azure 服务总线的功能。 Deferring messages as well.
延迟消息也是如此。 But it would require a different approach than batch processing.
但它需要一种不同于批处理的方法。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.