Apache Kafka - Resetting the last seen epoch of partition. Why?

Question

We use Kafka Mirror Maker Version 1 to mirror data between Kafka Clusters. I know that MM1 is deprecated but it is a solid pice of software that does exactly what we need. We use it out of a dedicated Kafka installation that is independent of the clusters between which we store data (which are running Kafka Version 2.6 by now).

We used Kafka 2.7.x for the "MirrorMaker-Kafka" and recently updated this one to 3.3.1. Since then we have a lot of the following messages in out MM log:

2022-12-01 15:07:45,368 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:12:45,373 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:22:45,388 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:37:45,394 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:42:45,398 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw

As you can see the message is repeated every 5 Minutens. We mirror several hundret partitions and the message is logged for many (maybe all) partitions.

As I could found out the message was introduced in the KAFKA-12257 fix in this commit .

Unfortunately, the meaning of the message is unclear to me. The constant repetition at a certain interval also makes me wonder. Possibly I still have weaknesses in my (Producer) configuration.

If someone could explain the phenomenon and know what measures I can take to improve it, I would be very pleased.

Answer 1

The message that you are seeing is related to the KAFKA-12257 fix, which was implemented to address an issue with the producer client not properly tracking the epoch of a topic partition. Specifically, the fix ensures that the producer client is able to keep track of the epoch of a partition by associating a unique identifier (the "topicId") with it.

In your case, the message is being logged every 5 minutes because the producer client is resetting the last seen epoch of a specific partition (MyTopic-5) to 87, since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw.

To ensure that the producer is able to properly track the epoch of a partition, you should review your producer configuration and ensure that the "topicId" is set properly. Additionally, you should also ensure that your producer client is configured to properly handle partition reassignment events, as this can cause the topicId to change.

Finally, if you are still having issues with the producer client not properly tracking the epoch of a partition, you may want to consider upgrading to a newer version of Kafka Mirror Maker, as MM1 is now deprecated.