如何使用其字段和基于时间的分区为 json 配置 kafka s3 接收器连接器？

Question

I have a json coming in like this:

{
    "app" : "hw",
    "content" : "hello world",
    "time" : "2018-05-06 12:53:04"
}

I wish to push to S3 in the following file format:

/upper-directory/$jsonfield1/$jsonfield2/$date/$HH

I know I can achieve:

/upper-directory/$date/$HH

with TimeBasedPartitioner and Topic.dir, but how do I put in the 2 json fields as well?

Answer 1

You need to write your own Partitioner to achieve a combination of TimeBased and Field Partitioners

That means make a new Java project, look at the source code for a reference point , build a JAR out of the project, and then copy the jar into kafka-connect-storage-common on all servers running Kafka Connect, which is picked up by the S3 connector. After you've copy the JAR, you will need to reboot the Connect process.

Note: there's already a PR that is trying to add this - https://github.com/confluentinc/kafka-connect-storage-common/pull/73/files