Kafka Producer (with multiple instance) writing to same topic

Question

I have a use case where messages are coming from a channel, which we want to push into a Kafka topic(multiple partitions) . In our case message order is important so we have to push the messages to topic in the order they are received which looks very straight forward if we have only one producer and single partition. In our case, for load balancing and scalability we want to run multiple instances for same producer but the problem is how to maintain order of messages .

Any thought or solution would be great helpful.

Even if I think to have single partition can it replicated to multiple brokers for availability and fault tolerance?

Answer 1

we have to push the messages to topic in the order they are received which looks very straight forward if we have only one producer and single partition

You can have multiple partitions in the topic with one producer and still have the order maintained if you provide key for your messages. All messages with the same key produced by a single producer are always in order.

---

When you say multiple producers , I assume that you are having multiple instances of your application running and that you are not creating multiple producers in the same JVM instance.

Since you said channel , I suppose that it is a network channel like Datagram channel, for example. In that case, I suppose that you are listening on some port and sending the received data to Kafka.

I do not see a point in having multiple producers in the same instance producing to the same topic, so it is better to have a single producer send all the messages and for performance you can tune the producer properties like batch.size , linger.ms etc.

To achieve fault tolerance, have another instance running in HA mode (fail-over mode), so that if this instance dies the other automatically picks up.

If it is a network channel, you can run multiple instances and open the socket with the option SO_REUSEADDR in StandardSocketOptions and this way you only one producer will be active at any point and new producer will become active once the active one dies.