Is it possible to stream AWS cloudwatch logs

Question

I know its is possible to stream CloudWatch Logs Data to Amazon Elasticsearch Service. It is documented here , But is it possible to stream the logs data to a custom AWS Glue Job, or to an EMR Job?

Answer 1

The way streaming of CloudWatch Logs (CWLs) to ElasticSearch (ES) works, is that AWS creates a lambda function for you. So CWLs will stream to lambda first, and the lambda will then upload the log events to ES.

For Glue, you don't need lambda function, as glue can get its streaming data from kinesis streams. So you would have to setup CWL subscription to kinesis stream. The stream would be used as a source in a streaming Glue job.

For EMR you could also just stream log events from CWL to kinesis in the same way as for Glue. But to read the stream data in EMR, you would probably have to use EMR Input Connector for Kinesis .