Logstash to elasticsearch. Keys with dots

Question

I'm facing a problem with logstash configuration. You can find my logstash configuration below.

Ruby filter removes every dot - "." from my fields. It seems that every works fine - the result of data filtration is correct but elasticsearch magically responds with: "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field name [/ConsumerAdminWebService/getConsumerTransactions.call] cannot contain '.'"} where getConsumerTransactions.call is one of my field key.

input {
  http_poller {
    urls => {
      uatBackend1 => {
        method => get
        url => "http://some-url/"
        headers => {
          Accept => "application/json"
        }
      }
    }
    request_timeout => 60
    # Run every 30 seconds
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
    metadata_target => "http_poller_metadata"
 }
}
filter {
  ruby {
    init => "
      def remove_dots hash
          new = Hash.new
          hash.each { |k,v|
              if v.is_a? Hash
                  v = remove_dots(v)
              end
              new[ k.gsub('.','_') ] = v
              if v.is_a? Array
                  v.each { |elem|
                      if elem.is_a? Hash
                          elem = remove_dots(elem)
                      end
                      new[ k.gsub('.','_') ] = elem
                  } unless v.nil?
              end
          } unless hash.nil?
          return new
      end
  "
  code => "
      event.instance_variable_set(:@data,remove_dots(event.to_hash))
  "
 }
}
output {
  elasticsearch {
    hosts => localhost
  }
}

I'm afraid that this line of code is not correct: event.instance_variable_set(:@data,remove_dots(event.to_hash)) - result data is somehow pinned to the event but the original data persists unchanged and is delivered to Elasticsearch api.

I suppose some clarifications are required here:

• I use ES version > 2.0 so dots are not allowed
• Ruby filter should replace dots with "_" and it works great - resulting data is fully correct however ES replies with mentioned error. I suspect that filter does not replace event data but simply adds a new filed to Event object. ES then still reads primal data not the updated one.

To be honest Ruby is a magic to me :)

Answer 1

If you're using the ES version 2.0 it could be a version issue where ES doesn't pick up fields which contains . dots.

According to this response in this thread :

Field names cannot contain the . character in Elasticsearch 2.0.

As a work around you might have to mutate (rename) your field names into something like _ or - instead of using the . dot. This ticket pretty much explains this issue, where as . dots can be used in the ES versions which are after 2.0. Hope it helps!