Logstash: Output to Elasticsearch gives errors with ruby but works with stdout

Question

The error is related to the ruby code, how do I fix this ?

Config:

input {
  beats {
    port => 5044
    ssl => false
  }
}

filter {
  if [type] == "apache" {
ruby {
code => " if event['message']
event['message'] = event['message'].gsub('\x','Xx')
event['message'] = event['message'].gsub('\x','XXx')
end
"
}

json {
      source => "message"
    }
}
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Logstash log:

{:timestamp=>"2016-07-01T13:23:30.475000+0100", :message=>"Connection refused", :class=>"Manticore::SocketException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:incall_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:inperform_request'", "org/jruby/RubyProc.java:281:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:inperform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch -transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/sniffer.rb:32:inhosts'", "org/jruby/ext/timeout/Timeout.java:147:in timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/sniffer.rb:31:inhosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:79:in reload_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:72:insniff!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in start_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:insynchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-outp ut-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in start_sniffing!'", "org/jruby/RubyKernel.java:1479:inloop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:in `start_sniffing!'"], :level=>:error} {:timestamp=>"2016-07-01T13:23:52.470000+0100", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn} {:timestamp=>"2016-07-01T13:23:52.470000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn} {:timestamp=>"2016-07-01T13:23:52.471000+0100", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash: :Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn} {:timestamp=>"2016-07-01T13:23:52.471000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn} {:timestamp=>"2016-07-01T13:23:53.471000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} {:timestamp=>"2016-07-01T13:23:53.472000+0100", :message=>"CircuitBreaker::Open", :name=>"Beats input", :level=>:warn} {:timestamp=>"2016-07-01T13:23:53.473000+0100", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker: :OpenBreaker, :level=>:warn} {:timestamp=>"2016-07-01T13:23:53.972000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}

Answer 1

The problem is not from the ruby code, but from your connection to Elasticsearch.

In your stacktrace:

"/opt/logstash/vendor/bundle/jruby/1.9/gems/ logstash-output-elasticsearch -2.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:in

Either your Elasticsearch instance is not started or can not be reached by logstash.

Answer 2

I resolved this by deleting the indexes from elasticsearch.

curl -XDELETE 'http://localhost:9200/filebeat-*'

Note that the ruby does not work. I had to change my log format to combined log to handle the exception.