Kafka Streams Left Join Not Producing Required Results

Question

I have a KStream with userClick and using userID as key also a KTable containing user details also using userID as key. Both KStream and KTable has some number of partitions, use same partitioning strategies and use same keys.

When i use left join between these two majority of the click events are not being matched with user details, there are some matched. But when i change KTable with GlobalKTable these missing matches disappear all required user clicks are enriched with user details.

What can cause this issue? Does using a KeyValueMapper when joining KStream and GlobalKTable resolve the issue in KStream to Ktable join? if so what can be the solution.

Edit: UserId is a Compacted topic and being generated by Confluent .net client, i have changed the default partition strategy to murmur2(Java client's default config).

Answer 1

1. Does using a KeyValueMapper when joining KStream and GlobalKTable resolve the issue in KStream to Ktable join?

   IMO if we using GlobalKTable then we'll lose Kafka ability to scale on user table.

2. What can cause this issue? Can you debug some userID cases in which the user data is not enriched? then check the userID partition number of both click stream and user table .