Logstash与elasticsearch输出：如何写入不同的索引？

Question

I hope to find here an answer to my question that I am struggling with since yesterday:

I'm configuring Logstash 1.5.6 with a rabbitMQ input and an elasticsearch output.

Messages are published in rabbitMQ in bulk format, my logstash consumes them and write them all to elasticsearch default index logstash-YYY.MM.DD with this configuration:

input {
  rabbitmq {
  host => 'xxx'
  user => 'xxx'
  password => 'xxx'
  queue => 'xxx'
  exchange => "xxx"
  key => 'xxx'
  durable => true
}

output {
  elasticsearch {
  host => "xxx"
  cluster => "elasticsearch"
  flush_size =>10
  bind_port => 9300
  codec => "json"
  protocol => "http"
  }
stdout { codec => rubydebug }
}

Now what I'm trying to do is send the messages to different elasticsearch indices.

The messages coming from the amqp input already have the index and type parameters (bulk format).

So after reading the documentation: https://www.elastic.co/guide/en/logstash/1.5/event-dependent-configuration.html#logstash-config-field-references

I try doing that

input {
  rabbitmq {
  host => 'xxx'
  user => 'xxx'
  password => 'xxx'
  queue => 'xxx'
  exchange => "xxx"
  key => 'xxx'
  durable => true
}

output {
  elasticsearch {
  host => "xxx"
  cluster => "elasticsearch"
  flush_size =>10
  bind_port => 9300
  codec => "json"
  protocol => "http"
  index => "%{[index][_index]}"
  }
stdout { codec => rubydebug }
}

But what logstash is doing is create the index %{[index][_index]} and putting there all the docs instead of reading the _index parameter and sending there the docs !

I also tried the following:

index => %{index}
index => '%{index}'
index => "%{index}"

But none seems to work.

Any help ?

To resume, the main question here is: If the rabbitMQ messages have this format:

{"index":{"_index":"indexA","_type":"typeX","_ttl":2592000000}}
{"@timestamp":"2017-03-09T15:55:54.520Z","@version":"1","@fields":{DATA}}

How to tell to logstash to send the output in the index named "indexA" with type "typeX" ??

Answer 1

If your messages in RabbitMQ are already in bulk format then you don't need to use the elasticsearch output but a simple http output hitting the _bulk endpoint would do the trick:

output {
    http {
        http_method => "post"
        url => "http://localhost:9200/_bulk"
        format => "message"
        message => "%{message}"
    }
}

Answer 2

So everyone, with the help of Val, the solution was:

• As he said since the RabbitMQ messages were already in bulk format, no need to use elasticsearch output, the http output to _bulk API will make it (silly me)

• So I replaced the output with this:

   output { http { http_method => "post" url => "http://172.16.1.81:9200/_bulk" format => "message" message => "%{message}" } stdout { codec => json_lines } } 

• But it still wasn't working. I was using Logstash 1.5.6 and after upgrading to Logstash 2.0.0 ( https://www.elastic.co/guide/en/logstash/2.4/_upgrading_using_package_managers.html ) it worked with the same configuration.

There it is :)

Answer 3

If you store JSON message in Rabbitmq, then this problem can be solved. Use index and type as field in JSON message and assign those values to Elasticsearch output plugin.

index => "%{index}" //INDEX from JSON body received from Kafka Producer document_type => "%{type}" } //TYPE from JSON body

With this approach , each message can have their own index and type.