Async. read from NetworkStream blocks for 60 seconds. (Rx)

Question

When there is no data in the stream and I try to read the stream blocks for 60 seconds. When there is some data the read completes as desired. How can I rewrite the following code so it can read only when stream.DataAvailable is true?

I think I need something like Observable.While(dataAvailableObserver, AsyncRead)..

    public static IObservable<byte[]> AsyncRead(this NetworkStream stream, int bufferSize)
    {
        return Observable.Create<byte[]>(
            o => Observable.Defer(() => AsyncReadChunk(stream, bufferSize))
                     .Repeat()
                     .Subscribe(dataChunk =>
                                    {
                                        if (dataChunk.Length > 0)
                                        {
                                            o.OnNext(dataChunk);

                                            return;
                                        }

                                        Debug.Assert(!stream.DataAvailable);

                                        o.OnCompleted();
                                    }, o.OnError, o.OnCompleted));
    }

    public static IObservable<byte[]> AsyncReadChunk(this NetworkStream stream, int bufferSize)
    {
        var buffer = new byte[bufferSize];

        return Observable.FromAsyncPattern<byte[], int, int, int>(stream.BeginRead, stream.EndRead)(buffer, 0, bufferSize)
            .Select(cbRead =>
            {
                Console.WriteLine("Data chunk received.");

                var dataChunk = new byte[cbRead];

                Buffer.BlockCopy(buffer, 0, dataChunk, 0, cbRead);

                return dataChunk;
            });
    }

What I found is to read in small bufferSizes as the bigger buffers cause the waiting of the buffer to be filled (like in my scenario where the incoming data is small packets).

Answer 1

Since you're using Defer, you have to check for available data within your defer logic. The simplest way would be to do the check within the AsyncReadChunk method like:

public static IObservable<byte[]> AsyncReadChunk(this NetworkStream stream, int bufferSize) 
{ 
    if (!stream.DataAvailable)
    {
        return Observable.Empty<byte[]>();
    }
    else
    {
        var buffer = new byte[bufferSize]; 

        return Observable.FromAsyncPattern<byte[], int, int, int>(stream.BeginRead, stream.EndRead)(buffer, 0, bufferSize) 
            .Select(cbRead => 

Answer 2

I am a little unsure as to if this will help, but this is the ToObservable() method I use to read streams.

public static class ObservableApmExtensions
{
    public static IObservable<byte> ToObservable(this FileStream source)
    {
        return source.ToObservable(4096, Scheduler.CurrentThread);
    }

    public static IObservable<byte> ToObservable(this FileStream source, int buffersize, IScheduler scheduler)
    {
        return Observable.Create<byte>(o =>
        {
            var initialState = new StreamReaderState(source, buffersize);
            var subscription = new MultipleAssignmentDisposable();
            Action<StreamReaderState, Action<StreamReaderState>> action =
                (state, self) =>
                {
                    subscription.Disposable = state.ReadNext()
                        .Subscribe(
                            bytesRead =>
                            {
                                for (int i = 0; i < bytesRead; i++)
                                {
                                    o.OnNext(state.Buffer[i]);
                                }
                                if (bytesRead > 0)
                                    self(state);
                                else
                                    o.OnCompleted();
                            },
                            o.OnError);
                };

            var scheduledAction = scheduler.Schedule(initialState, action);
            return new CompositeDisposable(scheduledAction, subscription);
        });
    }

    private sealed class StreamReaderState
    {
        private readonly int _bufferSize;
        private readonly Func<byte[], int, int, IObservable<int>> _factory;

        public StreamReaderState(Stream source, int bufferSize)
        {
            _bufferSize = bufferSize;
            _factory = Observable.FromAsyncPattern<byte[], int, int, int>(source.BeginRead, source.EndRead);
            Buffer = new byte[bufferSize];
        }

        public IObservable<int> ReadNext()
        {
            return _factory(Buffer, 0, _bufferSize);
        }

        public byte[] Buffer { get; set; }
    }
}

I have not tried it with a NetworkStream, but it sounds like you could swap the check

if (bytesRead > 0)

to

if (source.DataAvailable)

You would also need to then change the source type to NetworkStream.

I think the scheduling in my code may help you with your blocking problems. Another alternative, if appropriate (I still dont really understand you problem), is you could use .Switch and create a nested observable.

This would mean when some data comes through you read it all off until it is done and then complete. Once you completed you start another sequence which will be any further data.

s1 --1-0-1-1|
s2          ---1-0-0-1-|        
s3                     ---0-0-1-0-1|
etc..
out--1-0-1-1---1-0-0-1----0-0-1-0-1|

s1, s2, s3 etc are the sequences that are a burst of data until stream.DataAvailable. Then these inner streams will complete, and kickoff a request (creating another inner observable sequence s2, s3, sN). The Switch (or Merge or Concat) all will be able to flatten these multiple sequences into a single one for your users to consume.

One other alternative that maybe easier to code is to have an IEnumerable<IObservable<byte>>. These are easy to create with just a method like this

public IEnumerable<IObservable<byte>> ConstantRead(string path)
{
    while (true)
    {
        yield return Observable.Using(
                () => new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.None),
                stream => stream.ToObservable(4096, Scheduler.ThreadPool));
    }
}

Change for your network stream requirements.

You then just flatten like this

_subscription = ConstantRead(@"C:\Users\Lee\MyFile.zip")
                .Concat()
                .Subscribe(...

I hope that helps.

Lee Campbell