没有消费者连接时，Kafka经纪人可以保留消息吗？

Question

I am trying to build a pub/sub application and I am exploring the best tools out there. I am currently looking at Kafka and have a little demo app already running. However, I am running into a conceptual issue.

I have a producer ( Java code):

    String topicName = "MyTopic;
    String key = "MyKey";

    Properties props = new Properties();
    props.put("bootstrap.servers", "localhost:9092,localhost:9093");
    props.put("acks", "all");
    props.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
    Producer<String, byte[]> producer = new KafkaProducer <String, byte[]>(props);

    byte[] data = <FROM ELSEWHERE>;
    ProducerRecord<String, byte[]> record = new ProducerRecord<String, byte[]>(topicName, key, data);

    try {
        RecordMetadata result = producer.send(record).get();
    }
    catch (Exception e) {
        // Nothing for now
    }
    producer.close();

When I start a consumer via the Kakfa command line tools:

kafka-console-consumer --bootstrap-server localhost:9092 --topic MyTopic

and then I execute the producer code, I see the data message show up on my consumer terminal.

However , if I do not run the consumer prior executing the producer, the message appears "lost". When I start the consumer (after executing the producer), nothing appears in the consumer terminal.

Does anyone know if it's possible to have the Kafka broker retain messages while there are no consumers connected? If so, how?

Answer 1

Append --from-beginning to the console consumer command to have it start consuming from the earliest offset. This is actually about the offset reset strategy which is controlled by config auto.offset.reset . Here is what this config means:

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (eg because that data has been deleted):

earliest : automatically reset the offset to the earliest offset

latest : automatically reset the offset to the latest offset

none : throw exception to the consumer if no previous offset is found for the consumer's group anything else: throw exception to the consumer.