log.retention 和 cleanup.policy - Spring Kafka

Question

I have the following configuration in my docker-compose.yml

kafka:
    image: 'confluentinc/cp-kafka:6.1.0'
    hostname: kafka
    container_name: kafka
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://127.0.0.1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_MINUTES: 5
      KAFKA_LOG_CLEANUP_POLICY: compact, delete
    depends_on:
      - zookeeper

I'm using Spring Kafka Binder Streams for creating Topics. Once the application is up, I can see the topics in kafka but I dont see log.retention.minutes or cleanup.policy for the config.

it works if I specify the configuration for a specific topic. Is it not possible to configure log retention and cleanup at broker level?

    @Bean
    public NewTopic crateEmployeeTopic() {
        return TopicBuilder
                .name(KafkaConfig.EMP_DATA_TOPIC)
                .partitions(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "90000")
                .config(TopicConfig.CLEANUP_POLICY_CONFIG, "compact, delete")
                .build();
    }

-- UPDATE --

I was using Kafdrop UI to check the topic config. It was showing configs when it was defined at topic level, but at the broker level it is not showing any config. I ran the describe command to double check and can now see cleanup.policy , but retention is still missing

$ kafka-topics --bootstrap-server localhost:9092 --describe -- topic  emp_data
Topic: emp_data    PartitionCount: 3      ReplicationFactor: 1    Configs: cleanup.policy=compact,delete
        Topic: emp_data   Partition: 0    Leader: 1       Replicas: 1     Isr: 1

Thank you for your time.

Answer 1

this command did the trick.

$ kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 1 --all --describe | grep retention

Answer 2

You can set broker config for log.retention.minutes as mentioned in here . Then all topics that config defaults to it.

cleanup.policy is also same. Read here for more information.