Kafka Cluster - Producer

Question

I have several questions about Kafka. If someone can help me by responding to one of them, i will be very thankful.

Thank you in advance :)

Q1) I know that partitions are split across Kafka Broker. But the split is based on what ?. For instance, if I have 3 brokers and 6 partitions, how to ensure that each broker will have 2 partitions ? How this split is currently made in Kafka ?

Q2) When a producer send a new message, what id the default format of the message ? Avro format ? How can I change this default format to another format which may be more suitable for example ?

Q3) I know that to configure the maximum size of a file (log segment) within a partition, I have to change the following configuration : log.segment.bytes (1G by default). But which configuration parameter, I have to change to increase/decrease the maximum size of a directory (ie a partition) ?

Q4) If a partition consider as the leader is dead, one of the follower partition will take the lead. What is the step, to elect the new leader ? (ie) How the election of a new leader is made of ?

Q5) What is the configuration parameter, that allow me to change the time between 2 persist on disk ? (persist data on disk - sequential write)

Q6) How the message is sent from the hard disk Head of a Kafka broker to a Kafka consumer ? What is the interaction between Kafka Broker and Zookeeper ? Is it Zookeeper which send the message to the consumer or Kafka Broker ?

Thank you in advance,

Answer 1

Q1: see How Partitions are split into Kafka Broker?

Q2: Brokers are agnostic to the message format -- they treat messages a plain byte arrays. Thus, it can handle any message format you want to have. The format is determined in your own code -- choose whatever you want and just provide the corresponding de/serializer to the producer/consumer.

Q3: Topics and thus partitions are either truncated after a configurable retention time passed ( log.retention.ms ) or if they grow beyond log.retention.bytes . Furthermore, topics can be compacted to avoid infinite growth. (cf. log.cleanup.policy )

Q4: For leader election Apache Zookeeper is used.

Q5: Don't understand the question.

Q6: ZK is only used to maintain metadata (which topics do exists for example). ZK is not involved in any actual data transfer of client-broker communication. Kafka uses its own network protocol. See the Kafka Wiki for more details: https://cwiki.apache.org/confluence/display/KAFKA/Index