不要坚持主题卡夫卡

Question

I have a use case for Kafka where I dont need the logs of some topics that are running on the server, some topics send a lot of messages while other not so much.

Currently the only "fix" I see was to set a pretty lax retention policy.

Thing is that I would like to keep the logs and persistent information for some other topics that are on the server, which are arguably more important to me.

What options do I have ?

Ideally I would like to discard logs and stored messages by topic.

Answer 1

You can set per-topic retention policies, yes.

Here's an example using kafka-topics :

kafka-topics.sh --zookeeper localhost:2181 \
                --alter \
                --topic MyTopic \
                --config retention.ms=1000

Answer 2

The retention settings can be set on a per-topic basis. You can do that either at creation time or alter the topic config once created. You can find the full list of topic configs on Kafka's website .

For topics you don't need to keep data, set retention limits that will delete data frequently.

On the other hand, for topics you need to keep the data, set retention limits that will keep the data for longer period of time.

For example, to set topic config when creating a topic (retention time=24hours):

kafka-topics.sh --zookeeper <ZK> --create --topic --partitions 1 --replication-factor 1 --config retention.ms=86400000

To update an existing topic:

kafka-configs.sh --zookeeper <ZK> --entity-type topics --entity-name <TOPIC> --alter --add-config retention.ms=86400000