使用kafka partitioner作为锁

Question

I have a spring boot application that sends messages into a Kafka topic, something like:

public class Message {
    
    @JsonProperty("customer_id")
    @SerializedName("customer_id")
    private String customerId;
    ...
}

For the sake of some business logic, I do not want to process 2 messages of the same customer in parallel. I want to use the Kafka partition order as some kind of lock. As far as I could understand, I can use the customerId as a key to my message and use a custom partitioner (implements Kafka partitioner) to direct the message with the same key to the same partition. And though I assume I am not the first one in history to seek this kind of functionality I've found it hard to find some kind of library/documentation for implementing such a thing. Can anyone guide me to an existing solution to my needs?

Answer 1

That is the behavior of the default partitioner; see its javadocs:

/**
 * The default partitioning strategy:
 * <ul>
 * <li>If a partition is specified in the record, use it
 * <li>If no partition is specified but a key is present choose a partition based on a hash of the key
 * <li>If no partition or key is present choose the sticky partition that changes when the batch is full.
 * 
 * See KIP-480 for details about sticky partitioning.
 */
public class DefaultPartitioner implements Partitioner {

Changing the number of partitions in the topic will change the destination partition so you might want to consider a custom partitioner.