通过 kafka mongo sink 连接器在 mongo 中的内联 json 数组对象中附加字段值

Question

publish first insertion :

{"Customer_id": 2, "transaction_id": "1", "idd": [999, 1111], "id": 1}

and then second one :

{"Customer_id": 2, "transaction_id": "2", "idd": [9, 10], "id": 1}

required result :

{"Customer_id": 2, "transaction_id": "2", "idd": [[9, 10] , [999, 1111]], "id": 1}

what i get:

{"Customer_id": 2, "transaction_id": "2", "idd": [9, 10] , "id": 1}

update strategy this update array not append but my expected result to append array

configuration :

document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy
value.projection.list=customer_id,transaction_id
value.projection.type=whitelist
writemodel.strategy=com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneBusinessKeyStrategy

how can we do this through mongo-kafka sink connector

Answer 1

Looks like you're trying to aggregate streaming data

Unless MongoDB somehow is able to provide this functionality via inserting the same document ID (which I doubt because how does it know that you only want to collect the idd field?), you would have to use Kafka Streams / KSQL or another stateful processing layer, to aggregate your values. The sink connector just forwards whatever it sees into the database ; the connector has no knowledge of previous records