Role of ZooKeeper in cluster?

Question

If I have cluster hosting 1 topic which has three partitions. So ZooKeeper(ZK) cluster hosting 3 broker instances.

Per mine understanding ,

1. Producer will interact with ZooKeeper to publish the message on broker.
2. ZK will decide internally which partition it needs to publish the message based on load on each broker instance. Broker will also interact with ZK to maintain offset per consumer instance
3. Similarly Consumer will interact with ZooKeeper to consume the message from broker. ZK will get the message out from right broker based on load.

But I got confused after reading below bold text from section Workflow of Queue Messaging / Consumer Group at kafka tutorial . Is mine understanding above wrong ? Based on below looks like producer/consumer does not interact directly with zookeeper. Is it otherway around where ZK interact with producer/consumer. If yes who(Zookeeper or broker) which broker instance message needs to be published or consumed ?

ZooKeeper service is mainly used to notify producer and consumer about the presence of any new broker in the Kafka system or failure of the broker in the Kafka system. As per the notification received by the Zookeeper regarding presence or failure of the broker then producer and consumer takes decision and starts coordinating their task with some other broker. Basically Apache Zookeeper sa distributed configuration and synchronization service

Answer 1

You seem to be very mixed up in that most of the things you think are done by Kafka brokers are actually done by the clients and that most of the things you think are done by Zookeeper are actually done by the brokers.

Kafka is a very scalable system because the clients do a lot of the processing. The parts not done by the clients are done by the brokers (and the special broker components called the Controller and the Coordinators). Zookeeper does very little other than store state and some configuration for the brokers (in a very reliable way)

Addressing your points:

1) Incorrect. The new Producer does not interact directly with ZooKeeper. Producer talks directly to the brokers to publish messages or make meta-data requests to find which broker is the leader for a partition it wants to publish to.

2) Incorrect. ZK does not "decide" anything. ZK is a replicated fault tolerant storage system that the brokers use to save information and state for the cluster. The decision on which partition to publish into is done in the Producer and depends on the key of the message being published and the client side partitioner algorithm. Partitions are not assigned based on load, they are assigned based on the key (or if the key is null) then using a round robin algorithm. The Broker will NOT interact with ZK to maintain offset per consumer instance. Consumers keep track of their own offsets and store them (occasionally, via offset commits) in the _consumer_offsets topic on the brokers.

3) Incorrect. New Consumer will NOT directly interact with ZooKeeper to consume the message from broker. ZK will NOT get the message out from right broker based on load. Consumers will talk directly to the brokers, join and leave consumer groups via RPCs sent to the brokers using the kafka protocol.

Answer 2

Kafka uses zookeeper for: 1. Leader selection: this is required for topic partitions where replication factor is higher. Idea is to choose a node as leader for partition which keeps track of offset management and replication to followers. ZK is used to elect the leader to make sure there is one available leader at all time. 2. Cluster Memberships: to manage the brokers 3. Topics: Manage the topic which exists on the cluster, number of partitions, number of replicas, replica locations, etc...

Consumer specific: You can choose to manage consumer offsets on ZK. Therefore ZK will manage the latest offset and members of the consumer group.

I hope this answers your query