Number of Messages Per Transaction in Kafka

Question

Are there guidelines on how many messages and/or partitions can be involved in a Kafka Producer Transaction before performance really starts to suffer?

Obviously, the more partitions are involved, the more coordination is required. But in Kafka Streams, for example, the default commit interval is 100ms. In that time, normally hundreds of messages can be processed by a Stream thread. And depending on the topology, that may involve many different output topics (and by extension, partitions). Does that mean that it's safe to push transactions with hundreds of messages and dozens of partitions?

I can't find anything about this in the documentation.

Answer 1

https://www.confluent.io/blog/transactions-apache-kafka/ I found this in Confluent's blog. Turns out that the overhead doesn't increase per message per transactin; rather, you actually increase throughput if you have more messages per transaction.

In fact, the average overhead per message decreases as you add more messages to the transaction.