Kafka 使用 JSONConverter 连接 MongoDB Kafka 连接器不起作用

Question

I'm trying to configure the Kafka connector to use mongoDB as the source and send the records into Kakfa topics. I've successfully done so, but I'm trying to do it with the JSONConverter in order to also save the schema with the payload.

My problem is that the connector is saving the data as follows:

{ "schema": { "type": "string" } , "payload": "{....}" }

In other words, it's automatically assuming the actual JSON is a string and it's saving the schema as String.

This is how I'm setting up the connector:

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
 "name": "newtopic",
 "config": { 
"tasks.max":1,
 "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector", 
 "key.converter":"org.apache.kafka.connect.json.JsonConverter", 
 "key.converter.schemas.enabled": "true",
 "value.converter":"org.apache.kafka.connect.json.JsonConverter", 
 "value.converter.schemas.enabled": "true",
 "connection.uri":"[MONGOURL]", 
 "database":"dbname", 
 "collection":"collname", 
 "pipeline":"[]", 
 "topic.prefix": "", 
 "publish.full.document.only": "true" 
 }}'

Am I missing something for the configuration? Is it simply not able to guess the schema of the document stored in MongoDB, so it goes with String?

Answer 1

There's nothing to guess. Mongo is schemaless, last I checked, so the schema is a string or bytes. I would suggest using AvroConverter or setting schema enabled to false

You may also want to try using Debezium to see if you get different results

Answer 2

The latest version 1.3 of the MongoDB connector should solve this issue.
It even provides an option for inferring the source schema.