Throttling with Logstash and scaling

Question

I am already using ELK stack for log processing. Now, I am planning to use Logstash for throttling of incoming data. But, not sure, How to scale up Logstash + throttling, if load increases.

Normally for scaling up Logstash, I was planing to run multiple simultaneous instances of Logstash, to share the load. But if throttling will be there then how throttling information will be shared among different instances of Logstash ? Or any other, completely different approach ?

Answer 1

Based on your additional comments, it sounds like the metrics{} filter would work. It allows you to count the number of events that are coming through logstash in a given period, and you could configure different "meter" types based on the different things you wanted to count.

Once you have them counted with that filter, you could have an output{} stanza that would alert you (eg email, hipchat, irc, pagerduty).