如何使用单个命令在 Apache Kafka 中创建主题列表

Question

As of now I am creating a topic one by one by using below command.

sh ./bin/kafka-topics --create --zookeeper localhost:2181  --topic sdelivery --replication-factor 1 --partitions 1

Since I have 200+ topics to be created. Is there any way to create a list of topic with a single command?

I am using 0.10.2 version of Apache Kafka.

Answer 1

This seems like more of a unix/bash question than a Kafka one: the xargs utility is specifically designed to run a command repeatedly from a list of arguments. In your specific case you could use:

cat topic_list.txt | xargs -I % -L1 sh ./bin/kafka-topics --create --zookeeper localhost:2181 --topic % --replication-factor 1 --partitions 1

If you want to do a "dry run" and see what commands will be executed you can replace the sh with echo sh .

Alternatively you can just make sure that your config files have default topic settings of --replication-factor 1 and --partitions 1 and just allow those topics to be automatically created the first time you send a message to them.

Answer 2

You could use Terraform or Kubernetes Operators which will not only help you create topics (with one command), but also manage them later.

But without custom solutions, you can use awk for this

Create a file

$ cat /tmp/topics.txt
test1:1:1
test2:1:2

Then use system function to execute the kafka-topics script, and parse the file

$ awk -F':' '{ system("./bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic=" $1 " --partitions=" $2 " --replication-factor=" $3) }' /tmp/topics.txt
Created topic "test1".
Created topic "test2".

And we can see the topics are created

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --list
test1
test2

Note: Hundreds of topics this quickly might overload Zookeeper, so might help to add a call to "; sleep 10" at the end.

As of Kafka 3.0, the Zookeeper flag is replaced by bootstrap-servers.