Confluent Kafka Connect：以同步方式运行多个接收器连接器

Question

We are using Kafka connect S3 sink connector that connect to Kafka and load data to S3 buckets.Now I want to load data from S3 buckets to AWS Redshift using Copy command, for that I'm creating my own custom connector.Use case is I want to load data that created over S3 to Redshift in synchronous way, and then next time S3 connector should replace the existing file and again our custom connector load data to S3. How can I do this using Confluent Kafka Connect,or my other better approach to do same task? Thanks in advance !

Answer 1

If you want data to Redshift, you should probably just use the JDBC Sink Connector and download the Redshift JDBC Driver into the kafka-connect-jdbc directory.

Otherwise, rather than writing a Connector, you could use Lambda to trigger some type of S3 event notification to do some type of Redshift upload

Alternatively, if you are simply looking to query S3 data, you could use Athena instead without dealing with any databases

---

But basically, Sink Connectors don't communicate between one another. They are independent tasks that are designed to initially consume from a topic and write to a destination, not necessarily trigger external, downstream systems.

Answer 2

You want to achieve synchronous behaviour from Kafka to redshift then S3 sink connector is not right option.

1. If you are using S3 sink connector then first put the data into s3 and then externally run copy command to push to S3. ( Copy command is extra overhead )
2. No customize code or validation can happen before pushing to redshift.
3. Redshift sink connector has come up with native jdbc library which is equivalent fast to S3 copy command.