简体   繁体   中英

does kafka broker needs to commit to disk before consumer can read

Once a producer sends message to broker, does broker needs to flush to disk before it can make it available to consumer? Or could it send directly from page cache memory before committing to disk.

We are trying to use kafka for equities market data where message rates for small periods of time can be very high, so writing/flushing to disk before exposing to consumer would add severe latency.

Kafka maintains consistency of data using replication. System would not be consistent ,If data is provided to consumer before writing/flushing and then leader of the partition is lost before flushing the data to disk.

Messages are available to consumer only after they have been written to a particular segment of a partition.

Messages are only allowed to be read from leader partition and only those messages can be read which has been replicated to all in-sync replicas.We will get an empty response if we will try to fetch those messages which are not replicated on in-sync replicas.

We should tune the following parameters so that in-sync replica would be in-sync with the leader so that data is available very quickly to be read by consumer.

replica.lag.time.max.ms , replica.fetch.wait.max.ms and replica.fetch.min.bytes

在此处输入图片说明

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM