Kafka Producer, Consumer, Broker in same host?

Question

Are there any downsides to running the same producer and consumer code for all nodes in the cluster? If there are 8 nodes in the cluster (8 consumer, 8 kafka broker, and 8 producers), would 8 producers be running at the same time in the cluster then? Is there a way to modify cluster so that only one producer runs at a time?

Answer 1

Kafka cluster is nothing but Kafka brokers running under a distributed consensus. Kafka cluster is agnostic about number of producers and consumers running around it. Producers and consumers are clients of the Kafka cluster. Producers will stream data to Kafka and consumers consume the data from Kafka. Within Kafka cluster data will be distributed within topics. Topics are sharded using partitions. If multiple consumers belong to the same consumer group consumers can work in a self healing fashion.

Is there a way to modify cluster so that only one producer runs at a time?

If you intend to run a single producer at certain point of time, you don't need to make any change within cluster.

Answer 2

Are there any downsides to running the same producer and consumer code for all nodes in the cluster?

The primary downsides here would be scalability and memory usage.

Producers and Consumers are not required to run on Brokers. Producers should be deployed where data is being generated (or running as separate hosts, like Kafka Connect workers).

Consumers should be scaled out independently based on the throughput and ordering guarantees that you need in your downstream systems.

There is nothing that says 8 brokers requires 8 producers and 8 consumers; partitions are what matters more

• If you have N partitions in a topic, you can only scale to N active consumers anyway, and infinitely many producers
• 8 brokers can hold lots of partitions for any given topic

Running a single producer is an implementation of your own code. The broker cannot force it.