grpc WriteAsync locking up server

Question

Grpc.Core 2.38.0

I have a collection of applications participating in inter process communication using grpc streaming. From time to time we've noticed a lockup and memory exhaustion (due to the lockup) in the server processes being unable to finish a call to IAsyncStreamWriter.WriteAsync(...)

Recent changes to grpc (.net) have changed the WriteAsync API to accept a CancellationToken , however this is not available in the Grpc.Core package.

Answer 1

A misconfigured grpc client accepting a stream can cause a deadlock. If a client does not dispose of the AsyncServerStreamingCall during error handling, the deadlock will occur on the server.

Example:

async Task ClientStreamingThread()
{
    while (...)
    {
        var theStream = grpcService.SomeStream(new());
        try
        {
            while (await theStream.ResponseStream.MoveNext(shutdownToken.Token))
            {
                var theData = theStream.ResponseStream.Current;
            }
        }
        catch (RpcException)
        {
            // if an exception occurs, start over, reopen the stream
        }
    }
}

The example above contains the misbehaving client. If an RpcException occurs, we'll return to the start of the while loop and open another stream without cleaning up the previous. This causes the deadlock.

"Fix" the client code by disposing of the previous stream like the following:

async Task ClientStreamingThread()
{
    while (...)
    {
        // important.  dispose of theStream if it goes out of scope
        using var theStream = grpcService.SomeStream(new());
        try
        {
            while (await theStream.ResponseStream.MoveNext(shutdownToken.Token))
            {
                var theData = theStream.ResponseStream.Current;
            }
        }
        catch (RpcException)
        {
            // if an exception occurs, start over, reopen the stream
        }
    }
}