Entity Framework Core async/wait deadlock when using “one-to-many” relationship and WebApi

Question

I'm encountering a deadlock when using asynchronous implementation of an EF Core provider.

Say I have the following models:

public class Player
{
    public string PlayerId { get; set;}

    public string Name { get; set;}

    public List<Game> Games { get; set;}
}

public class Game
{
    public string GameId { get; set; }

    public string PlayerId { get; set; }
    public Player Player { get; set;}
}

Now I want to run the following query:

ctx.Players
.Include(p => p.Games)
.Where(p => p.PlayerId == "123")
.Select(p => new {
    PlayerId = p.PlayerId,
    Games = p.Games.ToList()
}).ToListAsync();

When I run this code via Console Application/XUnit test it works as excepted... But when I run it via ASP.Net WebApi it get into deadlock and never ends...

I used ConfigureAwait(false) all the way down in order to prevent these kind of situations but it seems that the problematic code is down underneath. I think that it might be under the System.Interactive.Async library which EFCore use - To be more specific it is under: https://github.com/Reactive-Extensions/Rx.NET/blob/develop/Ix.NET/Source/System.Interactive.Async/ToAsyncEnumerable.cs#L72 there is a call to " Result " which actually blocks the execution thread.

Does anyone encountered this behavior, maybe there is some workaround?

Notice that if I don't load the "Games" entities then everything also works fine...

Edit: Added the StackTrace: (See the call to ToEnumerable )

Not Flagged 21672   5   Worker Thread   grpc 0 (cq 0) System.Interactive.Async.dll!System.Linq.AsyncEnumerable.**ToEnumerable_**<NG.Admin.BL.DAL.Entities.Game> Normal
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext)
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)
mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Threading.Tasks.Task.GetResultCore(bool waitCompletionNotification)
mscorlib.dll!System.Threading.Tasks.Task.Result.get()
System.Interactive.Async.dll!System.Linq.AsyncEnumerable.ToEnumerable_<NG.Admin.BL.DAL.Entities.Game>(System.Collections.Generic.IAsyncEnumerable<NG.Admin.BL.DAL.Entities.Game> source)
mscorlib.dll!System.Collections.Generic.List<NG.Admin.BL.DAL.Entities.Game>.List(System.Collections.Generic.IEnumerable<NG.Admin.BL.DAL.Entities.Game> collection)
System.Core.dll!System.Linq.Enumerable.ToList<NG.Admin.BL.DAL.Entities.Game>(System.Collections.Generic.IEnumerable<NG.Admin.BL.DAL.Entities.Game> source)
[Lightweight Function]
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.SelectAsyncEnumerable<Microsoft.EntityFrameworkCore.Storage.ValueBuffer, NG.Admin.BL.Api.Player>.SelectAsyncEnumerator.MoveNext(System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task.TrySetResult(bool result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult(bool result)
NG.Data.Spanner.EF.dll!NG.Data.Spanner.EF.Query.Internal.SpannerAsyncQueryingEnumerable<Microsoft.EntityFrameworkCore.Storage.ValueBuffer>.SpannerAsyncEnumerator.MoveNext(System.Threading.CancellationToken cancellationToken) Line 55
[Resuming Async Method]
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task.TrySetResult(bool result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult(bool result)
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.SelectAsyncEnumerable<Microsoft.EntityFrameworkCore.Storage.ValueBuffer, Microsoft.EntityFrameworkCore.Storage.ValueBuffer>.SelectAsyncEnumerator.MoveNext(System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task.TrySetResult(bool result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult(bool result)
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable.AsyncEnumerator.MoveNext(System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task.TrySetResult(bool result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult(bool result)
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable.AsyncEnumerator.BufferlessMoveNext(bool buffer, System.Threading.CancellationToken cancellationToken)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.TrySetResult(System.__Canon result)
Microsoft.EntityFrameworkCore.Relational.dll!System.Threading.Tasks.TaskExtensions.Cast.AnonymousMethod__0(System.Threading.Tasks.Task t)
mscorlib.dll!System.Threading.Tasks.ContinuationTaskFromResultTask.InnerInvoke()
mscorlib.dll!System.Threading.Tasks.Task.Execute()
mscorlib.dll!System.Threading.Tasks.Task.ExecutionContextCallback(object obj)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot)
mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution)
mscorlib.dll!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task task, bool taskWasPreviouslyQueued)
mscorlib.dll!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task task, bool taskWasPreviouslyQueued)
mscorlib.dll!System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task task, bool needsProtection)
mscorlib.dll!System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task completedTask, bool bCanInlineContinuationTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult(object result)
NG.Data.Spanner.EF.dll!NG.Data.Spanner.EF.Storage.Internal.SpannerRelationalCommand.ExecuteAsync(Microsoft.EntityFrameworkCore.Storage.IRelationalConnection connection, string executeMethod, System.Collections.Generic.IReadOnlyDictionary<string, object> parameterValues, bool closeConnection, System.Threading.CancellationToken cancellationToken) Line 41
[Resuming Async Method]
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult(object result)
NG.Data.Spanner.EF.dll!NG.Data.Spanner.EF.Storage.Internal.SpannerRelationalCommand.ExecuteAsync(NG.Data.Spanner.EF.Storage.Internal.IOBehavior ioBehavior, Microsoft.EntityFrameworkCore.Storage.IRelationalConnection connection, string executeMethod, System.Collections.Generic.IReadOnlyDictionary<string, object> parameterValues, bool closeConnection, System.Threading.CancellationToken cancellationToken) Line 128
[Resuming Async Method]
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Data.Common.DbDataReader>.SetResult(System.Data.Common.DbDataReader result)
NG.Data.Spanner.dll!NG.Data.Spanner.SpannerCommand.ExecuteDbDataReaderAsync(System.Data.CommandBehavior behavior, System.Threading.CancellationToken cancellationToken) Line 67
[Resuming Async Method]
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Google.Cloud.Spanner.V1.ResultSet>.SetResult(Google.Cloud.Spanner.V1.ResultSet result)
NG.Data.Spanner.dll!NG.Data.Spanner.SpannerConnection.RunQuery(string commandText) Line 118
[Resuming Async Method]
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Google.Cloud.Spanner.V1.ResultSet>.SetResult(Google.Cloud.Spanner.V1.ResultSet result)
Google.Api.Gax.Grpc.dll!Google.Api.Gax.Grpc.ApiCallRetryExtensions.WithRetry.AnonymousMethod__0(Google.Cloud.Spanner.V1.ExecuteSqlRequest request, Google.Api.Gax.Grpc.CallSettings callSettings)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask)
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations()
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree()
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.TrySetResult(System.__Canon result)
mscorlib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.SetResult(System.__Canon result)
Grpc.Core.dll!Grpc.Core.Internal.AsyncCall<Google.Cloud.Spanner.V1.ExecuteSqlRequest, Google.Cloud.Spanner.V1.ResultSet>.HandleUnaryResponse(bool success, Grpc.Core.Internal.ClientSideStatus receivedStatus, byte[] receivedMessage, Grpc.Core.Metadata responseHeaders)
Grpc.Core.dll!Grpc.Core.Internal.CallSafeHandle.StartUnary.AnonymousMethod__0(bool success, Grpc.Core.Internal.BatchContextSafeHandle context)
Grpc.Core.dll!Grpc.Core.Internal.CompletionRegistry.HandleBatchCompletion(bool success, Grpc.Core.Internal.BatchContextSafeHandle ctx, Grpc.Core.Internal.BatchCompletionDelegate callback)
Grpc.Core.dll!Grpc.Core.Internal.CompletionRegistry.RegisterBatchCompletion.AnonymousMethod__0(bool success)
Grpc.Core.dll!Grpc.Core.Internal.GrpcThreadPool.RunHandlerLoop(Grpc.Core.Internal.CompletionQueueSafeHandle cq, Grpc.Core.Profiling.IProfiler optionalProfiler)
Grpc.Core.dll!Grpc.Core.Internal.GrpcThreadPool.CreateAndStartThread.AnonymousMethod__0()
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart()

Answer 1

So after a very long debugging I found the problem and did some work around to solve it. I'll try to explain:

I'm building EF Core provider to the Google Spanner db . For this I'm using Google Spanner DotNet API - although it is not yet public and very unmature API (actually it is just auto-generated code).

This API uses the gRPC API and what I saw is that on the first async call with this API, the gRPC thread (The gRPC has its own thread pool implementation) is used and will be used for the next execution calls - since the continuations will run only on this thread from the gRPC thread pool.

EFCore calls the ToEnumerable for the "Games" list which use the System.Interactive.Async lib. On the the ToEnumerable implementation it actually calls " Result " which blocks the execution thread and waits for the result. Because the gRPC thread is used and wait for the execution to be finished we have a deadlock...

My workaround was to use the following code:

GrpcEnvironment.SetCompletionQueueCount(1);

This way the continuations is allowed to run on other threads from the gRPC thread pool.

This is only a workaround and should be used. However it help solving the issue in the meantime...

The real solution should probably be in the EFCore usage of the System.Interactive.Async lib, in regarding to the ToEnumerable... (It should be pure async).

Edit: If it might be relevant to anyone: Just released an EF Core provider for the Google Spanner db: https://github.com/NoGame/NG.Data.Spanner

Answer 2

Your issue is that you are attempting to materialize Games synchronously and the projected object asynchronously. Note the following:

.Select(p => new {
    PlayerId = p.PlayerId,
    Games = p.Games.ToList() // sync over async is blocking
}).ToListAsync();

What you would expect to work (in regular EF for instance) is:

.Select(p => new {
    PlayerId = p.PlayerId,
    Games = p.Games // let the provider handle the enumeration materialization
}).ToListAsync();

But according to your comments that doesn't actually return the games on the anonymous projected object. The issue, it seems then, is that EF Core seems to not let you get away with naive assignment (you should update the question to indicate that). I wonder if this isn't related to how .Include() works in EF Core. I am not familiar enough with the core version of EF to offer much help beyond telling you the "cause," but essentially the issue is that you are waiting to get an enumerator from a source that is waiting to materialize based on that enumeration.