Expiring the messages in Kafka Topic

Question

We are using Apache Kafka perform load test in our dev environment.
In our Linux box where we have installed confluent kafka,have limited space hence to perform load test we have added retention.ms property to the topic.

Idea is to remove the message from the topic after it is consumed by the consumer.

I have tried

kafka-topics --zookeeper localhost:2181 --alter --topic myTopic --config retention.ms=10000

it didn't work hence we re-created the topic and tried below option.

kafka-configs --alter --zookeeper localhost:2181 --entity-type topics --entity-name myTopic -add-config retention.ms=10000 

After running the process for few hours the broker is shutting down because of space constraint.

What other options i can try from Topic as well as from broker standpoint to keep expiring the messages reliably and claiming back the disk space for long running load test.

Answer 1

You can define the deletion policy based on the byte size in addition to the time.

The topic configuration is called retention.bytes and in the documentation it is describes as:

This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the "delete" retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes.

You can set it together with retention.ms and whatever limit (bytes or time) will be reached first, the cleaning is triggered.

Answer 2

This might be because your Log cleaner threads might not had triggered.

You did not provide much info of how much data is accumulated on the topics. But it might not be in GB's.

Log cleaner threads will be cleaning on the completed log segments. Default size of segment is 1 GB.

Modify your topic configuration segment.bytes to less value if you are expecting huge load.

or modify the configuration segment.ms to 1 min or 10 mins as per your requirement.

This should create the segments and based on your log retention time, cleaner threads will clean the older completed segments.