TPL Dataflow - Conditional loops

Question

Currently I'm working on a pipeline data flow, where each stage except stage 1 is a async running consumer and producer. I have objects "flowing" through my pipeline, which reference items. In Stage 3 I would like to create a loop and buffer all objects, that meet a special condition (Stage Loop).

If new objects come in (Stage 3) while there are other objects currently buffered (Stage Loop), I would like to check if they match in their referencing item and if so post those to the BufferBlock of Stage Loop.

The question is, how can I check the referencing item of all objects in Stage Loop from within Stage 3?

The pipeline kinda looks like this:

Incoming objects ->  
  BufferBlock1 -> Parsing (Stage2) ->  
  BufferBlock2 -> Processing (Stage3) ->
  BufferBlock3 -> Stage Loop ->  
    Back to BufferBlock 2

Answer 1

You really don't need that many BufferBlock 's in your chain. The TPL Dataflow contains a TransformBlock , which encapsulates the BufferBloсk and ActionBlock logic, and have an output block for handled messages.

As for the loop, you can link the blocks between each other with static extension method , so this could be looks like

stage2.LinkTo(stage3, CheckForExistingProcessing);
stage2.LinkTo(stage4);

Jere stage4 is a queue for messages which didn't pass the check and must be handled in a loop. You can setup additional ActionBlock , or, maybe, simply use TransformBlock to send messages again to appropriate stage. I think that you can also introduce the retry check as some messages probably couldn't be processed at all so somewhat reasons.

Also, as you've said that you have async logic, you probably should SendAsync messages rather than Post them (you can also use the overload with CancellationToken ):

// asynchronously wait for a sending with resending attempts
await stage1.SendAsync(m);
// asynchronously wait for a sending with resending attempts with possible cancellation
await stage2.SendAsync(m, token);

Post method is synchronous and drops messages if they aren't accepted by target, comparing the SendAsync method which tries to deliver message even if target cannot accept it right now.