What is the best approach to getting data into S3 for Elasticsearch and RabbitMQ?

Question

In my company we developed a few games for which for some games the events are being sent to either Elasticsearch and others to RabbitMQ. We have a local CLI which grabs the data from both, compiles the messages into compressed (Gzip) JSON files after which another CLI converts them to SQL statements and throws them into a local SQL Server. We want now to scale up but the current setup is painful and nowhere near real-time for analysis.

I've recently built an application in Python which I was planning to publish to a docker container in AWS. The script grabs data from Elasticsearch, compiles into small compressed JSONS and publishes to an S3 bucket. From there the data is ingested into Snowflake for analysis. So far I was able to get the data in quite quickly and looks promising as an alternative.

I was planning to do something similar with RabbitMQ but I wanted to find an even better alternative which would allow this ingestion process to happen seamlessly and help me avoid having to implement within the python code all sorts of exception calls.

1. I've researched a bit and found there might be a way to link RabbitMQ to Amazon Kinesis Firehose. My question would be: How would I send the stream from RabbitMQ to Kinesis?

2. For Elasticsearch, what is the best way to achieve this? I've read about the logstash plugin for S3 ( https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html ) and about logstash plugin for kinesis ( https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kinesis.html ). Which approach would be ideal for real-time ingestion?

Answer 1

My answer will be very theotic and need to be adapted tested in real world and adapted to your use case. For a near realtime behaviour, I would use logstash

• with elasticsearch input and a short cron. this post can help https://serverfault.com/questions/946237/logstashs-elasticsearch-input-plugin-should-be-used-to-output-to-elasticsearch
• S3 output (support gzip)
• maybe jdbc output to your DB
• RabbitMq output plugin

You can create more scallable archi by output to RabbitMQ and use other pipeline to listen to the queue and execute other tasks.

• From logstash ES -> Rabbit MQ
• From logstash RabbitMQ -> SQL
• From logstash RabbitMQ -> Kinesis
• From logstash RabbitMQ -> AWS
• etc....