[英]MongoDB Kafka Sink Connector doesn't process the RenameByRegex processor
I need to listening events from a Kafka Topic and Sink to a collection in MongoDB.我需要从 Kafka Topic 和 Sink 监听事件到 MongoDB 中的集合。 The message contains an nested object with an id property, like in the example above.该消息包含一个带有 id 属性的嵌套 object,如上例所示。
{
"testId": 1,
"foo": "bar",
"foos": [{ "id":"aaaaqqqq-rrrrr" }]
}
I'm trying to rename this nested id to _id with RegExp我正在尝试使用 RegExp 将此嵌套 id 重命名为 _id
{
"connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
"topics": "test",
"connection.uri": "mongodb://mongo:27017",
"database": "test_db",
"collection": "test",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
"value.projection.list":"testId",
"value.projection.type": "whitelist",
"post.processor.chain": "com.mongodb.kafka.connect.sink.processor.DocumentIdAdder, com.mongodb.kafka.connect.sink.processor.field.renaming.RenameByRegex",
"field.renamer.regexp": "[{\"regexp\":\"\b(id)\b\", \"pattern\":\"\b(id)\b\",\"replace\":\"_id\"}]"
}
And the result of a config/validate is 500 Internal Server Error
, with that message:配置/验证的结果是500 Internal Server Error
,并带有该消息:
{
"error_code": 500,
"message": null
}
I missing something or is a issue?我错过了什么或者是一个问题?
I think all you want is Kafka Connect Single Message Transform (SMT) and more precisely ReplaceField
:我认为您想要的只是Kafka Connect Single Message Transform (SMT) ,更准确地说是ReplaceField
:
Filter or rename fields within a Struct or Map.过滤或重命名 Struct 或 Map 中的字段。
The following will replace id
field name with _id
:以下将用_id
替换id
字段名称:
"transforms": "RenameField",
"transforms.RenameField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.RenameField.renames": "id:_id"
In your case, before applying the above trasnformation you might also want to Flatten
foos
:在您的情况下,在应用上述转换之前,您可能还需要foos
Flatten
"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "."
and finally apply the transformation for renaming the field:最后应用转换来重命名字段:
"transforms": "RenameField",
"transforms.RenameField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.RenameField.renames": "foos.id:foos._id"
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.