如何配置从 SQS 获取消息并将它们移动到 Kafka 主题的 Kafka 连接器？

Question

I have a use case where I want to move messages from SQS to a Kafka topic. The framework to be used is SpringBoot. So, whenever I run my code it should start moving the messages. I searched for some articles but there were very few. I am looking for some boilerplate code to start with that follow the best practices and how to proceed further.

Thanks in advance.

Answer 1

You need to make yourself familiar with Enterprise Integration Patterns and its Spring Integration implementation.

To take messages from AWS SQS you would need to use an SqsMessageDrivenChannelAdapter from a Spring Integration for AWS extension. To post records into an Apache Kafka topic you need a KafkaProducerMessageHandler from the spring-integration-kafka module.

Then you wire everything together via an IntegrationFlow bean in your Spring Boot configuration.

Of course you can use Spring Cloud for AWS and Spring for Apache Kafka directly. Choice is yours, but better to follow best practice and start developing really an integration solution.

Answer 2

Apache Kafka offers multiple ways to ingest data from different sources eg Kafka Connect , Kafka Producer , etc., and we need to be careful while selecting specific components of Kafka by keeping certain things in mind such as retry mechanism , scalability , etc.

The best solution, in this case, would be to use Amazon SQS Source Connector to ingest data from AWS SQS into Kafka topic and then write your consumer application to do whatever is necessary with the stream of records of that particular topic .