MassTransit - UsePartitioner - 多个消费者

Question

We recently ran into an issue where users submit a request multiple times and because we're using the unit of work pattern inside of our consumers, those multiple requests would create duplicate records inside of our database.

We saw that there was a UsePartitioner method that we can add so that messages are partitioned on an ID that we set, and this would make the consumer wait until a message with that partitioned ID is done before starting on the next. This seems to work just fine locally with my docker setup where I'm only running one container per service. However, I noticed that when we deploy this to our other environments, we're still having that issue with duplicate records trying to be generated. I can't think of what else it could be unless maybe because our other environments have multiple containers/consumers running, that maybe the partitioning only happens on the single consumer and not shared? Or is there an additional setting that we're missing?

I should also add that we are using kubernetes. In our dev environment, we have 4 pods running so all 4 pods have an instance of this consumer.

    public class TestConsumerDefinition : ConsumerDefinition<TestConsumer>
    {
        public TestConsumerDefinition()
        {
            ConcurrentMessageLimit = 20;
        }

        protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator,
            IConsumerConfigurator<TestConsumer> consumerConfigurator)
        {
            var partitioner = consumerConfigurator.CreatePartitioner(ConcurrentMessageLimit.Value);

            consumerConfigurator.Message<TestMessage>(
                x =>
                    x.UsePartitioner(partitioner, m =>
                        $"{m.Message.DrugId}-{m.Message.PatientId}"));
        }
    }

Thank you.

Answer 1

First, there isn't a facility to partition across load balanced consumers on separate instances. You could build your own distributed lock, but...

The best approach would be to ensure your consumer logic is idempotent. Either using an upsert, or checking if the data already exists before adding it. Or, for extra credit, add the appropriate database-level constraint to prevent duplicates (either using a unique constraint or index).

The reason being, even a partitioner isn't going to prevent two requests a second apart from having the same data. So idempotent operations are important when dealing with distributed system.