Values from array to Kafka Topic

Question

I use Kafka Connect with Mongo as a source. In my case I need send to consumer data in one row. For example - I have collection like this:

{
    "_id" : "sdasd",
    "client_id" : "11",
    "device_id" : "11aa11",
    "contacts" : [ 
        {
            "contact_id" : "1",
            "contact_name" : "FirstName LastName",
            "contact_numbers" : [ 
                {
                    "contact_num" : "+4912222222",
                }
            ]
        },
        {
            "contact_id" : "2",
            "contact_name" : "FirstName2 LastName2",
            "contact_numbers" : [ 
                {
                    "contact_num" : "+4911111111",
                }
            ]
        }
    ]
}

And I configured my Connector with

curl -X POST -H "Content-Type: application/json" --data '
  {"name": "mongo-source-contacts",
   "config": {
     "tasks.max":"1",
     "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
     "output.format.value":"schema",
     "output.schema.value":"{\"name\":\"MongoExchangeSchema\",\"type\":\"record\",\"namespace\":\"com.mongoexchange.avro\",\"fields\":[{\"name\": \"client_id\",\"type\": \"string\"},{\"name\": \"device_id\",\"type\": \"string\"}, {\"name\": \"contacts.contact_name\",\"type\": \"string\"}]}",
     "schema.compatibility": "NONE",
     "key.converter":"org.apache.kafka.connect.storage.StringConverter",
     "value.converter":"io.confluent.connect.avro.AvroConverter",
     "value.converter.schema.registry.url":"http://localhost:8081",
     "connection.uri":"mongodb://localhost:27017/replicaSet=globaldb",
     "publish.full.document.only": true,
     "topic.prefix":"t_cb",
     "topic.creation.default.partitions"        : 4,
     "topic.creation.default.replication.factor": 1,
     "database":"testdb",
     "collection":"contactbook_test"
}}' http://localhost:8083/connectors -w "\n"

In output.schema.value I read fields that I need. Is it possible to get data in topic as records for each key? For example values for this fields: 11 11aa11 FirstName LastName +4912222222 But now I get client_id, device_id and array with contacts. Thanks for attention!

Answer 1

I'm not sure I understand the question. You've got one record "_id" . The Mongo connector doesn't parse anything more than that to "drill down" into specific fields, let alone get only the keys/values within an array. That being said, your output value schema should match the collection and include an array type. Worth pointing out that dots are not valid in Avro field names

If you want to flatten the array into multiple contact objects, consume the Connect data on your own, and write out data to another topic, using Kafka Streams for example