Kafka S3 Sink基本疑惑

Question

Do I really need to use confluent (CLI maybe)? Can I write my custom connector?
How can I write my first Kafka Sink? How to deploy them?

For now, let's assume we have the following details:

Topic: curious.topic
S3 bucket name: curious.s3
Data in the topic: Text/String
My OS: Mac

Answer 1

You start at the documentation for S3 Sink, looking over the configuration properties, and understand how to run Connect itself and deploy any connector (use the REST API); no, confluent CLI is never needed.

You don't need to "write your own sink" because Confluent already has an S3 Sink Connector. Sure, you could fork their open-source repo, and compile it yourself, but that doesn't seem to be what you're asking.

You can download the connector using different command confluent-hub .

Note: pinterest/secor does the same thing, without Kafka Connect.