如何通过 user_id 对 Kafka 中的主题进行分区？

Question

I'm building a web app backend with SpringBoot and I have to use Kafka for sending messages. I want to have a topic for example "testTopic" and I want to produce there some messages from differents users to send the messages later to differents machines.

If the user A sends a message to his machine and user B sends a message to his machine. How can I differentiate who has sent which message and to which machine it should arrive?

I've read about Kafka topic partitions but I don't know if Im doing it well in my code.

Here I'm building my topic

    @Bean
public NewTopic kafkaExampleTopic() {
    return TopicBuilder.name("TestTopic").partitions(1).build();
}

Here I'm sending data to that topic

    @Bean
CommandLineRunner commandLineRunner(KafkaTemplate<String, String> kafkaTemplate) {
    return args -> {
        kafkaTemplate.send("TestTopic", String.valueOf(MessageBuilder.withPayload("Hello kafka testTopic uno con key 1")
                .setHeader(KafkaHeaders.MESSAGE_KEY, "1").build()));
        kafkaTemplate.send("TestTopic", String.valueOf(MessageBuilder.withPayload("Hello kafka testTopic uno con key 2")
                .setHeader(KafkaHeaders.MESSAGE_KEY, "2").build()));
    };
}

And this is my listener

    @KafkaListener(topics = "TestTopic", groupId = "exampleGroupId")
public void listenWithHeaders(
        @Payload String message,
        @Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition) {
    System.out.println(
            "Received Message: " + message
                    + "from partition: " + partition);
}

Thank you so much guys!

Answer 1

Topic Partitions need to be decided ahead of time.

For example, if you have numeric ids, you could define a topic with ten partitions, then create your own Partitioner class that will route every number into the partition based on its leading digit (ids 1, 10, 15, etc all go to partition 1). If you use hexadecimal values (such as UUID), maybe a topic with 16 partitions (af, 0-9). Alphanumeric-lowercase - 36, and so on.

By default, Kafka's DefaultPartitioner will perform a Murmur2 hash modulo-d by the number of topic partitions. With that, it's possible that ids 5 and 7, for example, could end up in the same partition. Depending on your consumer's needs, that might not be what you want.

Consumers are what run on the different machines. The partitions shouldn't matter except to know that consumers of the same group cannot be assigned the same partitions (if you only have one partition, only one consumer per group can read it).