Message produced to kafka but consumers does not receive “some” of the messages

Question

There are 4 bootstrap servers and I'm producing the message using bellow code

producer.send(new ProducerRecord<>(topic, partitionNumber, key, message, headers), 
    (metadata, exception) -> {
        if (exception == null) {
            // the record was successfully sent
            synapseLog.auditLog(String.format(SUCCESS_LOG, metadata.offset(), 
                metadata.partition(), key, messageSeq, topic, SUCCESS));
        } else {
            synapseLog.auditError(String.format(FAILED_LOG, key, messageSeq, 
                FAILED, exception.getMessage(), topic, message));
        }
    }
);

SUCCESS_LOG was printed when producing the message but consumers did not receive the message (there are 2 consumers with different group id's).'

in producer properties acks = 1

I have tried running a another test consumer with a different group id. but it didn't get the message as well, probably the message is not in the kafka.

What could be the issue here.

Thanks in advance.

Answer 1

With acks=1 , lost messages are almost inevitable over a period of time. It is one likely cause of your issue, though there could be other explanations - for example it's possible to write consumers in a way that loses messages, (eg processing in a separate thread and committing before messages have been processed).

In short, the issue with 1 ack is that when you get a successful response it means that the leader of the partition has the record, but followers do not, and if the leader is restarted then any messages that have not been consumed by the minimum insync replicas at the broker end are definitely lost. This is intentional, acks=all is far more robust but has a performance penalty so Kafka allows you to choose what is more important - high throughput on minimal kit or guaranteed delivery.

I wrote a post about this at https://chrisg23.blogspot.com/2020/02/kafka-acks-configuration-apology.html?m=1 which has more detail about what is going on, and hopefully is of interest (though to be honest it is a bit long-winded)

Answer 2

I notice that in ProducerRecord you have specified the partition. I have experienced in the past that that causes problems when you are producing to a topic partition but consuming by subscribing to the topic in general and not to a partition. Try consuming from that particular partition. You can do that by "Manual Partition Assignment" using assign(Collection) method in Consumer(from Kafka docs).