ELK Stack using Docker on EC2 Container Service

Question

I am contemplating setting up an ELK (ElasticSearch, LogStash and Kibana) stack on AWS using Docker images. But I am unsure about performance and persistent storage.

If I just deploy the docker images to the EC2 Container service with my configuration, then I guess I need to also point to a place for persistent storage for both LogStash and ElasticSearch. Is S3 storage fast enough, or does that even matter when I am talking about logs. I am pretty sure I can live with some minutes delay on the indexing, but using Kibana, I would like to get data reasonably fast.

Is this a viable solution for a production setup with a couple of gigs worth of logs daily. I expect the log volume to rise once we see the value of this and start logging more to get more insight.

So:

1. Is it fast enough to use S3 for storage of log files?

2. Is it a viable solution for a production site that produces 5+ gigs of data a day?

Answer 1

You might take a look at AWS Elasticsearch Service . It's Elastic Search and Kibana as a service on AWS that you don't have to manually manage. I've just started using it for application-level events that my (desktop app) users are voluntarily reporting, and it's been really useful.