多个分区上的 Kafka 消费者

Question

My consumer does not get every messages at time. I have Kafka cluster with 3 brokers (3 servers) with 3 partitions for topic and replication factor 3.

Topic: my-topic       Partition: 0    Leader: 2       Replicas: 2,1,3 Isr: 3,1,2
Topic: my-topic       Partition: 1    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
Topic: my-topic       Partition: 2    Leader: 1       Replicas: 1,3,2 Isr: 1,3,2

I have consumer in Java and I set max poll records on 50000 fetch bytes configs on 50MB. Application polls every minute. When I send 10 messages to topic "my-topic", consumer does not give me all messages but only some of them and the rest in next run. Messages are produced by script during my applicatin sleep. Do you think it can be caused by partitions that poll method gives me only messages from server/partition which response first and the rest in next run?

Consumer:

Map<String, Object> configurations = new HashMap<>();
configurations.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, servers);
configurations.put(ConsumerConfig.ALLOW_AUTO_CREATE_TOPICS_CONFIG, "true");
configurations.put(ConsumerConfig.GROUP_ID_CONFIG, groupId ;
configurations.put(ConsumerConfig.CLIENT_ID_CONFIG, groupId);
configurations.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
configurations.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
configurations.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
configurations.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, "52428800");
configurations.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, "52428800");
configurations.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "3600000");
configurations.put(JsonDeserializer.TRUSTED_PACKAGES, "my.package.model");
configurations.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "50000");

consumer = new KafkaConsumer<Object, Object>(configurations);
consumer.subscribe(Collections.singletonList("my-topic"));

while(true) {
    ConsumerRecords<Object, Object> records = consumer.poll(Duration.ofMillis(10000));

    if(records.count() > 0) {
        LOGGER.debug("records count: {}", records.count());
        handleMessages(records);
        consumer.commitSync();
    }
    sleep(60000);
}

In handle method I log messages. Client and ts (timestamp) are data in message. Consumer gave me in first only 3 messages with similiar offset (I would say messages from one server/partition) after one minute sleep gave me the rest with 2 different offsets (I would say two others servers/partitions).

2021-11-23 08:27:14.851 [DEBUG] --- records count: 3
2021-11-23 08:27:14.853 [DEBUG] --- offset=1175, client=test-27, ts=1637652419417
2021-11-23 08:27:14.857 [DEBUG] --- offset=1176, client=test-28, ts=1637652419418
2021-11-23 08:27:14.860 [DEBUG] --- offset=1177, client=test-29, ts=1637652419418

2021-11-23 08:28:14.924 [DEBUG] --- records count: 7
2021-11-23 08:28:14.925 [DEBUG] --- offset=232304, client=test-20, ts=1637652419406
2021-11-23 08:28:14.929 [DEBUG] --- offset=232305, client=test-21, ts=1637652419407
2021-11-23 08:28:14.933 [DEBUG] --- offset=232306, client=test-24, ts=1637652419411
2021-11-23 08:28:14.937 [DEBUG] --- offset=1141, client=test-22, ts=1637652419408
2021-11-23 08:28:14.941 [DEBUG] --- offset=1142, client=test-23, ts=1637652419410
2021-11-23 08:28:14.944 [DEBUG] --- offset=1143, client=test-25, ts=1637652419414
2021-11-23 08:28:14.949 [DEBUG] --- offset=1144, client=test-26, ts=1637652419415

Does anyone knows what I do wrong or missed some parameter in config and how to repaire it?

Thanks

Answer 1

This is normal, Kafka doesn't guarantee you get all messages in one batch.
You would not typically sleep between polls as this can cause the client to timeout. Instead you rely on the poll duration preventing busy spinning.