AWS CloudWatch 日志到 Kafka

Question

I want to push AWS CloudWatch Logs to Kafka and trying to find out the best architecture to achieve this.

One approach I was thinking of is to -> Subscribe to log group events on cloud watch by selecting log group and clicking on Action ->Stream to AWS Lambda and select the lambda that will stream data to Kafka.

So I might have to write some Java/Python function to write logs to Kafka from Lambda. However, I am not sure whether this is the right design.

Answer 1

Have you considered using CloudWatch Logs Source Connector for Kafka Connect? It should be pretty straightforward to use and doesn't require any code writing (unfortunately, I only used jdbc connectors so I can't tell for sure how hard it is to configure it).