Kafka Connect S3 Sink 添加元数据

Question

I am trying to add the metadata to the output from kafka into the S3 bucket.

Currently, the output is just the values from the messages from the kafka topic.

I want to get it wrapped with the following (metadata): topic, timestamp, partition, offset, key, value example:

{
    "topic":"some-topic",
    "timestamp":"some-timestamp",
    "partition":"some-partition",
    "offset":"some-offset",
    "key":"some-key",
    "value":"the-orig-value"
}

note: when I am fetching it throw a consumer it fetched all the metadata. as I wished.

my connector configuration:

{  
 "name" : "test_s3_sink",   
 "config" : {     
     "connector.class" : "io.confluent.connect.s3.S3SinkConnector",
     "errors.log.enable" : "true",     
     "errors.log.include.messages" : "true",
     "flush.size" : "10000",
     "format.class" : "io.confluent.connect.s3.format.json.JsonFormat",
     "name" : "test_s3_sink",
     "rotate.interval.ms" : "60000",
     "s3.bucket.name" : "some-bucket-name",
     "storage.class" : "io.confluent.connect.s3.storage.S3Storage",
     "topics" : "some.topic",
     "topics.dir" : "some-dir"
   }
 }

Thanks.

Answer 1

Currently, the output is just the values from the messages from the kafka topic.

Correct, this is the documented behavior. There's a setting for including the key data that you're missing, if you wanted that, as well, but no settings to get the rest of the data.

For the record timestamp, you could edit your producer code to simply add that as part of your records. (and everything else, for that matter, if you're able to query for the next offset of the topic every time you produce)

For Topic and Partition, those are part of the S3 file, so whatever you're reading the files with should be able to parse out that information; the offset value is also part of the filename, then add the line number within the file to get the (approximate) offset of the record.

---

Or, you can use a Connect transform such as this archive one that relocates the Kafka record metadata (except offset and partition) all into the Connect Struct value such that the sink connector will then write all of it to the files

https://github.com/jcustenborder/kafka-connect-transform-archive

---

Either way, ConnectRecord has no offset field , a SinkRecord does, but I think that's too late in the API for transforms to access it