[英]TPL Dataflow Consumer to Process Multiple Items at a time
I have a requirement to iterate through a large list and for each item call a web service to get some data. 我需要遍历一个大列表,对于每个项目,都需要调用一个Web服务来获取一些数据。 However, I want to throttle the number of requests to the WS to say no more than 5 concurrent requests executing at any one time. 但是,我想限制对WS的请求数量,以使在任何一次执行的并发请求不超过5个。 All calls to the WS are made using async/await
. 对WS的所有调用都使用async/await
。 I am using the TPL dataflow BufferBlock
with a BoundedCapacity
of 5. Everything is working properly but what I am noticing is that the consumer, which awaits the WS call, blocks the queue until its finished resulting in all the requests in the bufferblock being performed serially. 我正在使用BoundedCapacity
为5的TPL数据流BufferBlock
。一切正常,但是我要注意的是,等待WS调用的使用者将队列阻塞,直到队列结束为止,导致该缓冲区块中的所有请求都按顺序执行。 Is it possible to have the consumer always processing 5 items off the queue at a time? 是否有可能让消费者总是一次处理5个项目? Or do I need to set up multiple consumers or start looking into action blocks? 还是我需要建立多个消费者或开始研究行动块? So in a nutshell I want to seed the queue with 5 items. 简而言之,我想为队列添加5个项目。 As one item is processed a sixth will take its place and so on so I always have 5 concurrent requests going until there are no more items to process. 当一项被处理时,第六项将被取代,依此类推,所以我总是有5个并发请求,直到没有更多项可以处理为止。
I used this as my guide: Async Producer/Consumer Queue using Dataflow 我以此为指导: 使用数据流的异步生产者/消费者队列
Thanks for any help. 谢谢你的帮助。 Below is a simplified version of the code 下面是代码的简化版本
//set up
BufferBlock<CustomObject> queue = new BufferBlock<CustomObject>(new DataflowBlockOptions { BoundedCapacity = 5 });
var producer = QueueValues(queue, values);
var consumer = ConsumeValues(queue);
await Task.WhenAll(producer, consumer, queue.Completion);
counter = await consumer;
//producer
function QueueValues(BufferBlock<CustomObject> queue, IList<CustomObject> values)
{
foreach (CustomObject value in values)
{
await queue.SendAsync(value);
}
queue.Complete();
}
//consumer
function ConsumeValues(BufferBlock<CustomObject> queue)
{
while (await queue.OutputAvailableAsync())
{
CustomObject value = await queue.ReceiveAsync();
await CallWebServiceAsync(value);
}
}
Your use of TPL Dataflow is rather strange. 您对TPL Dataflow的使用很奇怪。 Normally, you'd move the consumption and processing into the flow . 通常,您将消耗和处理移入流程 。 Append a TransformBlock
to call the webservice. 附加一个TransformBlock
来调用Web服务。 Delete ConsumeValues
. 删除ConsumeValues
。
ConsumeValues
executes sequentially which is fundamentally not what you want. ConsumeValues
按顺序执行,这根本不是您想要的。
Instead of BoundedCapacity
I think you rather want MaxDegreeOfParallelism
. 取而代之的BoundedCapacity
我觉得您更希望MaxDegreeOfParallelism
。
You should be using ActionBlock
with MaxDegreeOfParallelism
set to 5. You may also want to set a BoundedCapacity
but that's for throttling the producer and not the consumer: 您应该使用MaxDegreeOfParallelism
设置为5的ActionBlock
。您可能还想设置一个BoundedCapacity
但这是为了限制生产者而不是消费者:
var block = new ActionBlock<CustomObject>(
item => CallWebServiceAsync(item),
new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 5,
BoundedCapacity = 1000
});
foreach (CustomObject value in values)
{
await block.SendAsync(value);
}
block.Complete();
await block.Completion;
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.