在 MongoDB Kafka 源连接器中配置 Pipeline

Question

I am using following configuration for source connector to filter and read only specific records with status as "PENDING" from MongoDB. Need to poll all records that are inserted/updated with PENDING status only. Source connector able to poll all records if pipeline is excluded. Can someone help me understand how to fix this and also is there a way to know that polling is completed like batch job is completed to orchestrate another process on kafka consumer?

name=mongo-source-demo
connector.class=com.mongodb.kafka.connect.MongoSourceConnector
tasks.max=1
# Connection and source configuration
connection.uri=mongodb://username:password@hostname:27017
database=test
collection=mongoDBtest
topic.prefix=mongodb.connector
poll.max.batch.size=1000
poll.await.time.ms=100000
publish.full.document.only=true
pipeline=[{"$match": { "Status" : "PENDING" }},{"$project":{"_id":1,"fullDocument":1}} ]

Answer 1

The problem is with the "" in the MongoDB operators. The correct pipeline configuration should be:

pipeline=[{$match: { "Status" : "PENDING" }},{$project:{"_id":1,"fullDocument":1}} ]