In Rx.Net how can I implement an observable feedback loop until the feedback is exhausted?

Question

I have the following API:

IObservable<IList<SqlDataRecord>> WriteToDBAndGetFailedSource(SqlConnection conn, IList<SqlDataRecord> batch)

It attempts to write the batch into database. If fails, the entire batch is returned, otherwise the returned observable is empty.

I also have a source producing the batches:

IObservable<IList<SqlDataRecord>> GetDataSource(string filePath, int bufferThreshold)

Now, I can combine them like this:

var failedBatchesSource = GetDataSource(filePath, 1048576)
  .Select(batch => WriteToDBAndGetFailedSource(conn, batch))
  .Merge(100);

This is going to write all the batches (at most 100 concurrently) and return an observable of failed batches.

What I really want is to feed the failed batches back into the source of the batches after a certain pause , could be while the original source is still producing batches. I could, of course, write something like this:

var failedBatchesSource = GetDataSource(filePath, 1048576)
  .Select(batch => WriteToDBAndGetFailedSource(conn, batch))
  .Merge(100)
  .Select(batch => WriteToDBAndGetFailedSource(conn, batch))
  .Merge(100);

But it is wrong, of course, because:

1. This breaks the requirement to have a pause before failed batches are handled again.
2. It may generate more than 100 concurrent write requests to the database.
3. It is like unwinding a for-loop with an unknown count of iterations - unproductive.

I can also break out of the observable monad once I have collected all the failures and start all over again inside a loop:

            var src = GetDataSource(filePath, 1048576);

            for (;;)
            {
                var failed = await src
                    .Select(batch => WriteToDBAndGetFailedSource(conn, batch))
                    .Merge(100)
                    .ToList();
                if (failed.Count == 0)
                {
                    break;
                }
                src = failed.ToObservable();
            }

But I wonder if I can do better while staying within the observable monad.

Answer 1

I think this might do the trick

public static IObservable<T> ProcessAll<T>(this IObservable<T> source, Func<T, IObservable<T>> processor, int mergeCount, TimeSpan failureDelay)
{
    return Observable.Create<T>(
        observer =>
            {
                var failed = new Subject<T>();

                return source.Merge(failed)
                        .Select(processor)
                        .Merge(mergeCount)
                        .Delay(failureDelay)
                        .Subscribe(failed.OnNext, observer.OnError, observer.OnCompleted);
            });
}

And use it like this:

GetDataSource(filePath, 1048576)
  .ProcessAll(batch => WriteToDBAndGetFailedSource(conn, batch), 100, TimeSpan.FromMilliseconds(500))
  .Subscribe();

ProcessAll is a horrible name but it's Friday night and I can't think of a better one.

Answer 2

Use an Observable.Buffer. That allows you to buffer until you have 100 records to send, or until X amount of time has passed.

Alternatively, an Observable.Interval will simply fire every X time span. You can add the concurrency limit when you handle the publish event.

Either of these should fire repeatedly so long as there objects to be published.