使用属性文件配置kafka主题

Question

I want to configure Kafka with some topics and their unique retention settings. so when I start Kafka it will load the server.properties file with those settings.

The only way I found was by starting Kafka, and then start and configure topics using kafka-topics.sh script.

Example:

bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config retention.ms=86400000

I am using Kafka over docker, so instead of setting the entry point to be the actual script that start Kafka, I need to create my own script which will start Kafka and run some shell commands to configure those topics. Also if I need to create those topics on my own, I need to start and create some logic regarding topics which already exist. and I prefer not to go that road.

Answer 1

need to create my own script which will start Kafka and run some shell commands to configure those topics

1. wurstmeister/kafka already supports KAFKA_CREATE_TOPICS environment variable for this purpose

2. It is recommended you use a second "init container", (or shell script around docker run/compose ) that waits for the other kafka container to start and be healthy, then do topic-creation. Related answer

3. Just use AdminClient API in your client code to create topics before running producers or consumers.