多个logstash节点上的Logstash聚合过滤器插件

Question

I'm interested in using the logstash aggregate filter plugin but I was wondering how this would work in my case where I have multiple logstash nodes .

Also do i need to set - filter workers to 1 ( -w 1 flag) - if I don't mind the events are possessed out of sequence?

Update

My use case demands aggregating logs generated by multiple services by a unique trace-id. I have no end event . Rather using a set window of say 3 seconds.

Answer 1

For the aggregate filter to work correctly you need to send the logs with the information you want to combine to the same logstash node.

If you are using a shipper, like Filebeat or any other, that sends events to multiple logstash nodes each node will aggregate only the events that it receives, you can't aggregate events from differents logstash nodes.

It is also recommended to set the workers to 1 because the main use for the aggregate filter is to get information from the events belonging to the same unique id (task id, job id, process id) and enrich a final event, if you use more than 1 worker, your end event could be processed before your start event, for example.

Answer 2

I feel this can be tackled by a combination of design and config. Thought I'd drop in a few thoughts on the topic - for what it's worth.

1. Aggregation is limited to single worker - for obvious reasons. Which forces your design to restrict aggregation at a single point in your information flow. Sounds like a hub/ spoke-wheel pattern to me
2. If you have multiple logstashes - you will need to conditionally route events that DO need aggregation to this central logstash. I'd go with adding a "to-be-aggregated" tags (for example) which can be used for flow control
3. On the aggregator logstash itself - either implicitly aggregate everything that comes in - or use the above tag to decide which one to do or NOT. Obviously, this copy will need to run with pipeline.workers=1 AND pipeline.java_execution=false ( see here ).
4. That leaves the rest down to config? Which I think leads you back to the specific filter plugin docs page

This is what I am doing. Hope this helps..