MongoDB Atlas Source Connector 单主题

Question

I am using Confluent MongoDB Atlas Source Connector to pull data from MongoDB collection to Kafka. I have noticed that the connector is creating multiple topics in the Kafka Cluster. I need the data to be available on one topic so that the consumer application can consume the data from the topic. How can I do this?

Besides, why the Kafka connector is creating so many topics? isn't is difficult for consumer applications to retrieve the data with that approach?

Answer 1

Kafka Connect creates 3 internal topics for the whole cluster for managing its own workload. You should never need/want external consumers to use these

In addition to that, connectors can create their own topics. Debezium for example creates a "database history topic", and again, this shouldn't be read outside of the Connect framework.

Most connectors only need to create one for the source to pull data into, which is what consumers actually should care about