Azure Durable Functions - OrchestrationTrigger continuously executing 1st awaited Activity

Question

I have created an Azure Durable Function following the structure of the Fan-out / Fan-In example, (using the recommended VS 2017 development, not the Azure Portal approach)

What I am experiencing is that very the first Azure Function that is called from the OrchestrationTrigger method is called over and over again, and never continues to the next statement. Now while I have read the Event Sourcing, Checkpointing, and Orchestrator Replay documentation and the Orchestrator Code Constraints , I am still a bit confused as debugging locally is very tricky due to the replay.

[FunctionName("ExportLogs")]
public static async Task ExportLogs([OrchestrationTrigger] DurableOrchestrationContext context)
{
    SinceDate since = context.GetInput<SinceDate>();
    var sinceDate = DateTime.Parse(since.since);

    var conversationInfos = await context.CallActivityAsync<IEnumerable<ConversationInfo>>("GetConversations", sinceDate);

    // !! I never get to this statement
    var tasks = new List<Task<CallInfo>>(conversationInfos.Count());
    foreach (var ci in conversationInfos)
    {
        Task<CallInfo> task = context.CallActivityAsync<CallInfo>("ProcessCall", ci);
        task.Start();
        var notused = task.ContinueWith(t => context.CallActivityAsync("PersistCallInStorage", t.Result));
        tasks.Add(task);
    }
    await Task.WhenAll(tasks);

    // Now fan-in and persist all results.
    var callInfos = tasks.Select(t => t.Result);
    var connectionString = ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString;
    var repo = new DwRepository(connectionString);
    repo.AddCallInfos(callInfos);
}

The first activity called is:

[FunctionName("GetConversations")]
public static IEnumerable<ConversationInfo> GetConversations([ActivityTrigger] DateTime sinceDate, TraceWriter log)
{
    var exportService = ConstuctExportService();
    var conversations = exportService.GetAllConversations(sinceDate);
    return conversations;
}

Now, exportService.GetAllConversations(sinceDate); is calling upon an SDK that is fetching some call information from a cloud service via REST. It does need to loop and fetch the data in pages. So what is happening is that this get called over and over and never continues on, even though I am awaiting the call.

Here is the output from the Function Runtime output window:

[10/23/2017 7:25:43 PM] Function started (Id=8e62e6bd-34a9-49b7-a814-3a0e9e5f9f8e)
[10/23/2017 7:25:43 PM] Executing 'ExportRscCallLogs' (Reason='', Id=8e62e6bd-34a9-49b7-a814-3a0e9e5f9f8e)
[10/23/2017 7:25:43 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'ExportRscCallLogs (Orchestrator)', version '' started. IsReplay: False. Input: (22 bytes). State: Started. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:25:44 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'GetConversations (Activity)', version '' scheduled. Reason: ExportRscCallLogs. IsReplay: False. State: Scheduled. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:25:44 PM] Function completed (Success, Id=8e62e6bd-34a9-49b7-a814-3a0e9e5f9f8e, Duration=114ms)
[10/23/2017 7:25:44 PM] Executed 'ExportRscCallLogs' (Succeeded, Id=8e62e6bd-34a9-49b7-a814-3a0e9e5f9f8e)
[10/23/2017 7:25:44 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'ExportRscCallLogs (Orchestrator)', version '' awaited. IsReplay: False. State: Awaited. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:25:44 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'GetConversations (Activity)', version '' started. IsReplay: False. Input: (23 bytes). State: Started. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:25:44 PM] Function started (Id=3c33ff3b-5fe5-4e6b-a7d4-ac6c3e146633)
[10/23/2017 7:25:44 PM] Executing 'GetConversations' (Reason='', Id=3c33ff3b-5fe5-4e6b-a7d4-ac6c3e146633)
[10/23/2017 7:25:47 PM] Function completed (Success, Id=3c33ff3b-5fe5-4e6b-a7d4-ac6c3e146633, Duration=2824ms)
[10/23/2017 7:25:47 PM] Executed 'GetConversations' (Succeeded, Id=3c33ff3b-5fe5-4e6b-a7d4-ac6c3e146633)
[10/23/2017 7:25:47 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'GetConversations (Activity)', version '' completed. ContinuedAsNew: False. IsReplay: False. Output: (435216 bytes). State: Completed. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:25:54 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'GetConversations (Activity)', version '' started. IsReplay: False. Input: (23 bytes). State: Started. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:25:54 PM] Function started (Id=d89b2fa4-f4d7-43e5-95f1-5cf436ebc85e)
[10/23/2017 7:25:54 PM] Executing 'GetConversations' (Reason='', Id=d89b2fa4-f4d7-43e5-95f1-5cf436ebc85e)
[10/23/2017 7:25:56 PM] Function completed (Success, Id=d89b2fa4-f4d7-43e5-95f1-5cf436ebc85e, Duration=1397ms)
[10/23/2017 7:25:56 PM] Executed 'GetConversations' (Succeeded, Id=d89b2fa4-f4d7-43e5-95f1-5cf436ebc85e)
[10/23/2017 7:25:56 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'GetConversations (Activity)', version '' completed. ContinuedAsNew: False. IsReplay: False. Output: (435216 bytes). State: Completed. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:26:05 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'GetConversations (Activity)', version '' started. IsReplay: False. Input: (23 bytes). State: Started. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.
[10/23/2017 7:26:05 PM] Function started (Id=fb240d9c-90f0-4e4d-86db-b3d2c175e0eb)
[10/23/2017 7:26:05 PM] Executing 'GetConversations' (Reason='', Id=fb240d9c-90f0-4e4d-86db-b3d2c175e0eb)
[10/23/2017 7:26:06 PM] Function completed (Success, Id=fb240d9c-90f0-4e4d-86db-b3d2c175e0eb, Duration=1177ms)
[10/23/2017 7:26:06 PM] Executed 'GetConversations' (Succeeded, Id=fb240d9c-90f0-4e4d-86db-b3d2c175e0eb)
[10/23/2017 7:26:06 PM] b350f681b17b4ac8a772fff8fdca3aad: Function 'GetConversations (Activity)', version '' completed. ContinuedAsNew: False. IsReplay: False. Output: (435216 bytes). State: Completed. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.

As you can see, the logs are showing that it is Executing 'GetConversations' over and over again, and it even reports that it was completed successfully each time. The IsReplay parameter is False - so I am not sure what's going on?

Any help would be greatly appreciated.

Answer 1

Thanks to Chris Gillum's efforts in identifying the problem on the posted issue . He discovered that my initial activity is

returning back a relatively large amount of data (482564 bytes, or 471 KB). Return values get serialized into queue messages, and Azure storage queues only support 64 KB messages . I suspect the runtime is not handling this gracefully and is resulting in it running continuously.