Does kafka consumer handle same topic keys if it consumes multiple topics?

Question

Let's say I have 2 kafka topics logins and logouts partitioned by username and having equal number of partitions . If I run a consumer group of 2 consumers consuming both topics, can I be sure that login and logout events of each user will be handled by the same consumer?

Answer 1

I don't think so at least with default configuration ... let me explain better ... Imagine you have 2 partitions for each topic so : logins topic has partitions 0 and 1 (let's call them lin0, lin1) logouts topic has partitions 0 and 1 (let's call them lout0, lout1)

We have both consumers in the same consumer group and they are called c1 and c2. Using subscribe() method with c1 could happen that it will get lin0 and lout1, while c2 could get lin1 and lout0. When a producer will write to logins with username = u1 (for example) it could go in lin0 with the message related to login of username u1 and this message will be consumed by c1. It can happen that on logout, the producer will writes messages with username = u1 to lout0 which is handled by c2 ! So login handled by c1 but logout by c2 ! That it's want you don't want of course ! This scenario happens due to subscribe() method which handles automatic partitions assignment and default partitioner which on the producer distributes messages in a round robin fashion to topic partitions.

On potential solution could be writing a custom partitioner to use on the producer so that with some criteria based on username, u1 will be always written on partition 0 for both topics (so lin0 and lout0). Then using assign() on the consumer side, so asking for a specific partition so for example c1 could ask to consumer exactly from lin0 and lout0. Of course you lose rebalancing feature provided by subscribe.