面临 Azure cosmos 多区域 AKS 更改提要处理器的问题

Question

We have our custom Change Feed processor deployed in single Region in AKS with 5 instances. Things were always running in single region fine. (Please note that each pod instance (feed processor) is assigned a unique [.WithInstanceName(new GUID)].

We recently moving to a multi region setup as following:

• EastUS AKS Cluster = 5 pods (5 feed processor with each unique instance name)
• WestUS AKS Cluster = 5 pods (5 feed processor with each unique instance name)

Now with the above setup, the result is not very consistent as sometimes after the AKS service deployment our feedprocessor stops recieving the events for some of the collections).

To fix this we need to eventually delete the lease collection and then everything starts working again.

We cannot go live with the above workaround, so need help to resolve the issue.

Here is the code snippet:

Container leaseContainer = cosmosClient.GetContainer(databaseName, leaseContainerName);
            changeFeedProcessor = cosmosClient.GetContainer(databaseName, sourceContainerName)
                .GetChangeFeedProcessorBuilder(processorName: sourceContainerName, async (IReadOnlyCollection<TContainer> changes, CancellationToken cancellationToken) => await onChangesDelegate(changes, cancellationToken))
                    .WithInstanceName($"{Guid.NewGuid()}")
                    .WithLeaseContainer(leaseContainer)
                    .Build();

where leaseContainerName = "container-lease"

Answer 1

The problem is that you are mixing instances. If you want each region to work independently (the same document change go to both groups of processors independently), set a different processorName .

When you define a cluster of machines with a particular processorName , lease and monitored containers, you define a Deployment Unit . The change feed events are distributed across those machines.

If you deploy 2 clusters but with the same values, then the 10 pods are now the same Deployment Unit, so the changes are spread across the 10 pods, meaning now that a particular change will land in 1 of the instances on one of the region but the other region will not see it.

You could set as processorName the region name for example:

Container leaseContainer = cosmosClient.GetContainer(databaseName, leaseContainerName);
            changeFeedProcessor = cosmosClient.GetContainer(databaseName, sourceContainerName)
                .GetChangeFeedProcessorBuilder(processorName: regionName, async (IReadOnlyCollection<TContainer> changes, CancellationToken cancellationToken) => await onChangesDelegate(changes, cancellationToken))
                    .WithInstanceName($"{Guid.NewGuid()}")
                    .WithLeaseContainer(leaseContainer)
                    .Build();