How to set transaction-id-prefix in app which has both consumer and producer-only transactions in Spring Cloud Stream

Question

For transactions that have been started by a listener container, we need to set the same transaction id prefix for all app instances. For producing-only transaction, we need to set a different value for each instance instead.

I'm using Spring Cloud Stream Kafka Binder in the app which has both types of transactions, and the property spring.cloud.stream.kafka.binder.transaction.transaction-id-prefix is used for creating a common transaction manager.

I wonder how to make it all work properly because it seems that you can't have it both ways at the same time.

Answer 1

That requirement is no longer necessary if your broker is 2.5 or later and you are using a currently supported version of spring-cloud-stream (3.2.x) and spring-kafka (2.8.x).

You must now use a unique prefix even for consumer-initiated transactions.

If you are using an older broker or an unsupported spring-kafka version that does not use EOSMode V2 (BETA) by default, you can configure two binders.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#transaction-id-prefix

This problem (different rules for transactional.id) has been eliminated when EOSMode.BETA is being used (with broker versions >= 2.5); see Exactly Once Semantics.