Result of task invocation on a grain in Orleans

Question

I apologize for the long question. I have been experimenting with Orleans to know about its various properties and these questions are logically under one umbrella.

---

The first test involved making request from client to a specific grain every 1 second while the grain takes 10 seconds to execute the requests. The code is this:

// client code
while (1)
{
    Console.WriteLine("Client giving another request");
    double temperature = random.NextDouble() * 40;
    var sensor = client.GetGrain<ITemperatureSensorGrain>(500);
    Task t = sensor.SubmitTemperatureAsync((float)temperature);
    Console.WriteLine(t.Status);
    Thread.Sleep(1000);
 }

// grain code
public Task SubmitTemperatureAsync(float temperature)
{
    long grainId = this.GetPrimaryKeyLong();
    Console.WriteLine($"{grainId} outer received temperature: {temperature}");
    Thread.Sleep(10000);

    Console.WriteLine($"{grainId} outer complete");
    return Task.CompletedTask;
}

The console output is:

Client giving another request
Task Status - WaitingForActivation
500 outer received temperature: 32.29987    <------------ print statement inside grain
Client giving another request     <--------------------- client continues
Task Status - WaitingForActivation  <------------------- client isn't blocked
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
Client giving another request
Task Status - WaitingForActivation
500 outer complete

As grains in Orleans are single threaded, only first request is invoked and rest requests are queued on the grain side. My questions for this part are: -

In normal C#, when an async method is called, it continues on the main thread till it hits the await statement when it starts the awaited expression as another Task and returns that Task. So, the caller is blocked till the await statement is hit. Similarly, here too the client should be blocked for 10s after which the first request to grain returns a Task. However, that doesn't happen. The client is continuing to schedule tasks without getting blocked.

• So, is the call to grains from client FireAndForget ?
• If yes, then how do they get back the Task object?
• Is there any kind of blocking involved when client makes a call to a grain object and runtime brings Task object back to the client?

---

The second test involved making request from a grain to a grain in which the second grain waits 10 seconds before returning. The code is this:

// client code
while (1)
{
    Console.WriteLine("Client giving another request");
    double temperature = random.NextDouble() * 40;
    var sensor = client.GetGrain<ITemperatureSensorGrain>(500);
    Task t = sensor.SubmitTemperatureAsync((float)temperature);
    Console.WriteLine("Client Task Status - "+t.Status);

    // make client sleep for a long time after the first request
    // because we don't want any more requests from the client
    Thread.Sleep(1000000000);
}

// outer-grain (ITemperatureSensorGrain) code
public async Task SubmitTemperatureAsync(float temperature)
{
    long grainId = this.GetPrimaryKeyLong();
    Console.WriteLine($"{grainId} outer received temperature: {temperature}");

    while(true)
    {
        Console.WriteLine("Grain sending another request");
        ITempBGrain sensor = this.GrainFactory.GetGrain<ITempBGrain>(400);
        // await sensor.SubmitTempBAsync(temperature);
        Task t = sensor.SubmitTempBAsync(temperature);
        Console.WriteLine("Grain Task Status - "+t.Status);
        Thread.Sleep(1000);
    }
}

// inner-grain (ITempBGrain) code
public Task SubmitTempBAsync(float temperature)
{
    long grainId = this.GetPrimaryKeyLong();
    Console.WriteLine($"{grainId} internal received temperature: {temperature}");
    Thread.Sleep(10000);
    Console.WriteLine($"{grainId} internal complete");
    return Task.CompletedTask;
}

The console output is:

Client giving another request
Client Task Status - WaitingForActivation
500 outer received temperature: 10.36764
Grain sending another request       <-------------- Outer grain prints
Grain Task Status - WaitingForActivation 
Grain sending another request  <----------- Inner grain doesn't print
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
Grain sending another request
Grain Task Status - WaitingForActivation
warn: Orleans.Runtime.CallbackData[100157]
      Response did not arrive on time in 00:00:30 for message: Request *cli/015ba7a5@d4cdc7ab->S127.0.0.1:30000:0*grn/6424EE47/000001f4 #17: . Target History is: <S127.0.0.1:30000:0:*grn/6424EE47/000001f4:>. About to break its promise.
Grain sending another request
Grain Task Status - WaitingForActivation

What I see here is something similar to what happened with client in the first experiment. So, those questions are still there. However, here there is one more strange thing happening. The console output of inner grain appears nowhere. Why is the inner grain not executing? If I enable the commented out line in outer-grain code, and await the inner grain task, the following output appears which seems valid.

Client giving another request
Client Task Status - WaitingForActivation
500 outer received temperature: 6.332514
Grain sending another request
400 internal received temperature: 6.332514
400 internal complete
Grain sending another request
400 internal received temperature: 6.332514

Answer 1

First Part
No, there is no blocking when a call to a grain is made. This post further clears what happens when a grain call is made.

Second Part
While it is correct that grains are single threaded, it is wrong to assume that every grain has its own thread in Orleans. As @Tseng says Orleans uses the async feature of .NET Core. It will process a grain until an async operation happens. Then it returns the thread to the thread-pool. This thread can be used by another grain to process data until the async operation is complete. When its complete, it resumes. its not necessary the same thread (but its the same context) Orleans uses the async feature of .NET Core. It will process a grain until an async operation happens. Then it returns the thread to the thread-pool. This thread can be used by another grain to process data until the async operation is complete. When its complete, it resumes. its not necessary the same thread (but its the same context) Orleans uses the async feature of .NET Core. It will process a grain until an async operation happens. Then it returns the thread to the thread-pool. This thread can be used by another grain to process data until the async operation is complete. When its complete, it resumes. its not necessary the same thread (but its the same context) . The first grain is blocking the thread giving no chance for the second grain to execute.

Thanks Tseng and Reuben Bond for making things clear.