我们如何将多个集合映射到 mongodb-sink-connector 中的多个主题？

Question

I am using a mongodb sink connector to read from some topics json data, and then write them into their respective collections in mongodb. But, when I give multiple collections with topic-collection mappings, the connector doesn't recognize the collection names, and instead create collections with the name same as the topics'. Below is the worker config:

name=sink-mongonew_Transaction_test4
connector.class=com.mongodb.kafka.connect.MongoSinkConnector
connection.uri=mongodb://****:27017
database=ec_ods
topics=t6,t5
collections=abc,def
collection.t6=abc
collection.t5=def
type.name=kafka-connect
key.ignore=true
document.id.strategy.abc=com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy
value.projection.type.abc=whitelist
value.projection.list.abc=ID
writemodel.strategy.abc=com.mongodb.kafka.connect.sink.writemodel.strategy.UpdateOneTimestampsStrategy
document.id.strategy.def=com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy
value.projection.type.def=whitelist
value.projection.list.def=ID
writemodel.strategy.def=com.mongodb.kafka.connect.sink.writemodel.strategy.UpdateOneTimestampsStrategy
delete.on.null.values=false
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

Answer 1

https://github.com/mongodb/mongo-kafka/blob/master/config/MongoSinkConnector.properties

I found a different way of using multiple collections in the above link, which uses overriding, and even allows the use of global as well as local attributes assignment for multiple collections at the same time.

Answer 2

This is the updated link.

https://github.com/mongodb/mongo-kafka/blob/master/config/MongoSinkConnector.properties

I guess the repo was updated.

Answer 3

Here a link to a proper documentation of the MongoDB Kafka sink connector https://github.com/mongodb-labs/mongo-kafka/blob/master/docs/sink.md

As per documentation, you should use the topic.override.%s.%s property

The overrides configuration allows for per topic customization of configuration. The customized overrides are merged with the default configuration, to create the specific configuration for a topic. For example, topic.override.foo.collection=bar will store data from the foo topic into the bar collection. Note: All configuration options apart from 'connection.uri' and 'topics' are overridable. string ""