kafka sink 连接器中无效 JSON 的错误处理

Question

I have a sink connector for mongodb, that takes json from a topic and puts it into the mongoDB collection. But, when I send an invalid JSON from a producer to that topic (eg with an invalid special character ") => {"id":1,"name":"\\"} , the connector stops. I tried using errors.tolerance = all, but the same thing is happening. What should happen is that the connector should skip and log that invalid JSON, and keep the connector running. My distributed-mode connector is as follows:

{
  "name": "sink-mongonew_test1",  
  "config": {
    "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
    "topics": "error7",
    "connection.uri": "mongodb://****:27017", 
    "database": "abcd", 
    "collection": "abc",
    "type.name": "kafka-connect",
    "key.ignore": "true",

    "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
    "value.projection.list": "id",
    "value.projection.type": "whitelist",
    "writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.UpdateOneTimestampsStrategy",

    "delete.on.null.values": "false",

    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter.schemas.enable": "false",

    "errors.tolerance": "all",
    "errors.log.enable": "true",
    "errors.log.include.messages": "true",
    "errors.deadletterqueue.topic.name": "crm_data_deadletterqueue",
    "errors.deadletterqueue.topic.replication.factor": "1",
    "errors.deadletterqueue.context.headers.enable": "true"
  }
}

Answer 1

Since Apache Kafka 2.0, Kafka Connect has included error handling options , including the functionality to route messages to a dead letter queue, a common technique in building data pipelines.

https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/

As commented, you're using connect-api-1.0.1.*.jar , version 1.0.1, so that explains why those properties are not working

Your alternatives outside of running a newer version of Kafka Connect include Nifi or Spark Structured Streaming