How to organize sequence of data processors with .net RX

Question

What is the best way to organize sequence of data processors with .net RX?
- a. Call methods on observable like observable.Do(log).Select(transformation).Do(work).Aggregate(someState)...
- b. Implement custom observers, if so - how to chain them
- c. Other option.. And also what is the best option to handle possible exceptions in observable itself (see my concerns above) and to handle exceptions inside Do, Select, etc (as I know the best practice is that Subscribers shouldn't throw).

Also I need sometimes to allow exceptions being returned as some elements of observable sequence without sequence being stopped (see this question Handling Exceptions in Reactive Extensions without stopping sequence )

Answer 1

This seems like you need a workflow instead of Rx. It seems based on your other questions* you are trying to take what looks very well suited to a ProducerConsumer queuing workflow scenario, and forcing it into Rx.

It looks like you

• want to Read from a Queue and block until a value is received, then resubscribe. Just use Queuing features of BlockingCollection. As values arrive they can be pushed into the collection from any thread.
• dont want to actually have a sequence of values, but a workflow that can fail on any given value, divert that and then process the next value. Just use a queue. Process each value and put the results into the next appropriate queue (Failure/Success) ->See Enterprise Integration Patterns , specifically Invalid Message Channel and Dead Letter channel.
• potentially process these values in parallel. See BlockingCollection.GetConsumingEnumerable or the Disruptor for a high performance implementation. With these tools you can have many produces and consumers. Sure you can do this with Rx, but it is just polling and tying up a thread doing so. I think it is better to be explicit about this kind of thing

I think your usage of Rx should pay for itself and you shouldn't find yourself fighting it (like any other technology or framework).

*other questions like Handling Exceptions in Reactive Extensions without stopping sequence and How to serialize Observables to the cloud and back