MongoDB Kafka 源连接器 - copy.exisintg.pipeline 配置

Question

I'm running a MongoDB Kafka source connector (official mongodb connector version 1.7.0), and defining both pipeline and copy.existing.pipeline properties in order to filter some columns- see below full source connector config.

The problem I'm having is that the copy.existing.pipeline seems to work and filter the columns during the copy.existing phase, but when the connector finishing and moves to the stream phase it seems like the pipeline property is ignored and I'm getting the documents without any filtering.

{
  "name": "connector1",
  "config": {
    "name": "connector1",
    "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
    "connection.uri": "mongodb://mongo:27017/admin?tls=true&tlsallowinvalidcertificates=true&tlsallowinvalidhostnames=true",
    "database": "myDB",
    "collection": "myCollection",
    "change.stream.full.document": "updateLookup",
    "poll.max.batch.size": 5000,
    "topic.prefix": "topic",
    "copy.existing": true,
    "copy.existing.allow.disk.use": true,
    "copy.existing.max.threads": 10,
    "output.format.value": "schema",
    "pipeline": "[{$project: {\"aql\": 0, \"wizard_expressions\": 0}}]",
    "copy.existing.pipeline": "[{$project: {\"aql\": 0, \"wizard_expressions\": 0}}]"
  }

Any advice here?

Answer 1

It seems like I had to filter the columns prefixed by fullDocument .

"pipeline": "[{$project: {" fullDocument .aql": 0, " fullDocument .wizard_expressions": 0}}]",

"copy.existing.pipeline": "[{$project: {" fullDocument .aql": 0, " fullDocument .wizard_expressions": 0}}]"