如何确定为什么并行运行的持久函数仍然很慢

Question

We have the following function which splits a task into separate functions (Fan Out)

[FunctionName("process-orchestration")]
public async Task ProcessOrchestrationAsync(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    var runProcessEvent = context.GetInput<RunProcessDto>();

    var processStartedAt = context.CurrentUtcDateTime;

    // 1. Initialise process

    var initialiseProcessResponseDto = await context.CallActivityAsync<InitialiseProcessResponseDto>
        ("initialise-process-activity", );

    // 2. Run process segment tasks in parallel

    var tasks = new List<Task<RunProcessSegmentResponseDto>>();
    foreach (var runProcessRequestDto in initialiseProcessResponseDto.RunProcessRequestDtos)
    {
        var subTask = context.CallActivityAsync<RunProcessSegmentResponseDto>("run-process-segment-activity", runProcessRequestDto);
        tasks.Add(subTask);
    }

    await Task.WhenAll(tasks);

    // 3. Aggregate results

    var runProcessSegmentResponseDtos = new List<RunProcessSegmentResponseDto>();

    foreach (var eachTask in tasks)
    {
        var taskResult = eachTask.Result;
        runProcessSegmentResponseDtos.Add(taskResult);
    }

    var aggregateProcessResultDto = new AggregateProcessResultDto()
    {
        RunProcessRequestDto = new RunProcessRequestDto()
        {
            NoOfDays = runProcessEvent.NoOfDays,
        },
        RunProcessSegmentResponseDtos = runProcessSegmentResponseDtos
    };
    var runProcessResponseDto = await context.CallActivityAsync<RunProcessResponseDto>("aggregate-process-result-activity", aggregateProcessResultDto);

    // 4. Finalise process

    runProcessResponseDto.ProcessStartedAt = processStartedAt;
    runProcessResponseDto.ProcessFinishedAt = context.CurrentUtcDateTime;
    await context.CallActivityAsync<RunProcessResponseDto>("finalise-process-activity", runProcessResponseDto);
}

This takes a date range and splits into segments

Each segment is then processed in parallel

After all segments are processed, the results are combined, this process is fine

However, when I look at the breakdown of the segments there are some segments that take a really long time to run, even though they dont have many days

Start date: 03/07/2022 
End date: 01/10/2022 
No of days: 90

Diagnostic started: 01/10/2022 10:02:48 GMT 
Diagnostic finished: 01/10/2022 11:41:51 GMT

Segment start date: 03/07/2022
Segment end date: 18/07/2022 
Segment started at: 01/10/2022 10:02:48
Segment finished at: 01/10/2022 11:41:36

Segment start date: 19/07/2022
Segment end date: 03/08/2022 
Segment started at: 01/10/2022 10:02:48
Segment finished at: 01/10/2022 11:37:26

Segment start date: 04/08/2022
Segment end date: 19/08/2022 
Segment started at: 01/10/2022 10:02:48
Segment finished at: 01/10/2022 11:35:59

Segment start date: 20/08/2022
Segment end date: 04/09/2022 
Segment started at: 01/10/2022 10:02:48
Segment finished at: 01/10/2022 11:17:13

Segment start date: 05/09/2022
Segment end date: 20/09/2022 
Segment started at: 01/10/2022 10:02:48
Segment finished at: 01/10/2022 10:19:05

Segment start date: 21/09/2022
Segment end date: 01/10/2022 
Segment started at: 01/10/2022 10:02:48
Segment finished at: 01/10/2022 10:14:03

I tried to scale the function up to be the max power premium function and the timings are no different (this was in place for the timings above)

I also added this to my host.json in case there are too many segments being run in parallel

"extensions": {
    "durableTask": {
      "hubName": "%HubName%"
    },
    "maxConcurrentActivityFunctions": 5,
    "maxConcurrentOrchestratorFunctions": 5
  },
  "functionTimeout": "-1"

Are there any considerations that I have missed when doing this?

Cheers

Paul

Answer 1

maxConcurrentActivityFunctions and maxConcurrentOrchestratorFunctions should be inside durableTask section.

"extensions": {
  "durableTask": {
    "hubName": "%HubName%",
    "maxConcurrentActivityFunctions": 5,
    "maxConcurrentOrchestratorFunctions": 5
  },
},
"functionTimeout": "-1"

I've faced the similar issue and at the end I used App Service Plan with a fixed set of VM instances and auto-scale rules.

Not very serverless but with more control over the resources that app uses.

Hosting plans