Logstash中的两个输出。 仅用于某些聚合

Question

I'm trying to specify a second output of logstash in order to save certain aggregated data only. No clue how to achieve it at the moment. Documentation doesn't cover such a case.

At the moment I use a single input and a single output.

Input definition ( logstash-udp.conf ):

input { 
    udp { 
        port => 25000
        codec => json
        buffer_size => 5000
        workers => 2
    }
}

filter {
  grok {
    match => [ "message", "API call happened" ]
  }

  aggregate {
    task_id => "%{example_task}"
    code => "
        map['api_calls'] ||= 0
        map['api_calls'] += 1
        map['message'] ||= event.get('message')
        event.cancel()
    "
    timeout => 60
    push_previous_map_as_event => true
    timeout_code => "event.set('aggregated_calls', event.get('api_calls') > 0)"
    timeout_tags => ['_aggregation']
  }
}

Output definition ( logstash-output.conf ):

output {
  elasticsearch {
    hosts => ["localhost"]
    manage_template => false
    index => "%{[@metadata][udp]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

What I want to achieve now? I need to add a second, different aggregation (different data and conditions) which will save all the not aggregated data to Elasticsearch like now however aggregated data for this aggregation would be saved to Postgres. I'm pretty much stuck at the moment and searching the web for some docs/examples doesn't help.

Answer 1

I'd suggest using multiple pipelines: https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

This way you can have one pipeline for aggregation and second one for pure data.