如何使用 Kafka 连接 s3 接收器连接器标记 S3 存储桶对象

Question

Is there any way we can tag the objects written in S3 buckets through the Kafka Connect S3 sink connector. I am reading messages from Kafka and writing the avro files in S3 bucket using S3 sink connector. When the files are written in S3 bucket I need to tag the files.

Answer 1

there is an API inside source code on the GitHub called addTags(), but it's now private and is not exposed to the connector client except this small config feature called S3_OBJECT_TAGGING_CONFIG which allows you to add start/end offsets as well as record count to s3 object.

 configDef.define(
      S3_OBJECT_TAGGING_CONFIG,
      Type.BOOLEAN,
      S3_OBJECT_TAGGING_DEFAULT,
      Importance.LOW,
      "Tag S3 objects with start and end offsets, as well as record count.",
      group,
      ++orderInGroup,
      Width.LONG,
      "S3 Object Tagging"
  );

If you want to add other/custom tags then answer is NO you cannot do it right now.

Useful feature would be to take the tags from the predefined part of an input document in Kafka but this is not available right now.