Logstash无法将输出发送到弹性搜索

Question

This question has already been asked here https://stackoverflow.com/questions/30283837/logstash-unable-to-send-output-to-elasticsearch but I can't see the actual answer. Can someone advise if there is a solution to this please as I am suffering the same problem.

I believe this is a system parameter causing the problem as I've run the same config on a different machine and it works fine. Anyone know what this might be?

I've looked everywhere to find an answer but can only find a firewall issue (so tried outside the firewall) and the time stamp being different between the twitter server and my machine (machine timing set correctly). Can anyone advise what's causing the below authorisation error (I've also checked the twitter app and settings are correct and working).

Logstash startup completed
←[33m {:exception=>Twitter::Error::Unauthorized, :backtrace=>["C:/logstash-1.5.1
/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/response.rb:2
1:in on_headers_complete'", "org/ruby_http_parser/RubyHttpParser.java:370:in<
<'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/
streaming/response.rb:16:in <<'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/ge
ms/twitter-5.12.0/lib/twitter/streaming/connection.rb:22:instream'", "C:/logst
ash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/clie
nt.rb:116:in request'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter
-5.12.0/lib/twitter/streaming/client.rb:36:infilter'", "C:/logstash-1.5.1/vend
or/bundle/jruby/1.9/gems/logstash-input-twitter-0.1.6/lib/logstash/inputs/twitte
r.rb:88:in run'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-core
-1.5.1-java/lib/logstash/pipeline.rb:176:ininputworker'", "C:/logstash-1.5.1/v
endor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:17
0:in `start_input'"], :level=>:warn}←[0m

I'm using the below twitter config:-

input {
twitter {
consumer_key => "xxxxxxxxxxxxxxxxxx"
consumer_secret => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
oauth_token => "xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
oauth_token_secret => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
keywords => ["xxxxxxxxxxx"]
full_tweet => true
}
}

output {
stdout { codec => dots }
elasticsearch {
protocol => "http"
host => "localhost"
index => "twitter"
document_type => "tweet"
template => "twitter_template.json"
template_name => "twitter"
}
}

I tried the same from outside the firewall and instead of authorisation issue I get a connection refused. So I believe warkolm you were correct but that's just left me with a different problem.

←[33mConnection refused - Connection refused {:exception=># Connection refused - Connection refused>, :backtrace=>["org/jruby/ext/socket/Ru
byTCPSocket.java:126:in initialize'", "org/jruby/RubyIO.java:853:innew'", "C:
/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streamin
g/connection.rb:16:in stream'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems
/twitter-5.12.0/lib/twitter/streaming/client.rb:116:inrequest'", "C:/logstash-
1.5.1/vendor/bundle/jruby/1.9/gems/twitter-5.12.0/lib/twitter/streaming/client.r
b:36:in filter'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-inpu
t-twitter-0.1.6/lib/logstash/inputs/twitter.rb:88:inrun'", "C:/logstash-1.5.1/
vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:1
76:in inputworker'", "C:/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-c
ore-1.5.1-java/lib/logstash/pipeline.rb:170:instart_input'"], :level=>:warn}←[
0m

This is even after trying a new set of twitter authorization codes.

If this same configuration is run on another developers machine it works fine so there is a config on my desktop that isn't set correctly (both machines are running the same setup windows 8.1 running ES 1.7.1 and LS 1.5.1).

Any thoughts on what setting is missing on my machine.

Thanks Leigh

Answer 1

The stack trace that you provided originates in the Twitter connection phase -- its's getting connect refused when trying to initiate a connection to stream.twitter.com:443. Since that's unlikely that the streaming api for twitter was down, you've probably got a firewall blocking you in some way.

Looking up stream.twitter.com, there are multiple A records for it... So it's possible that whomever runs your firewall missed an address or something like that. It could be that the other developer has a host file entry to force it to a specific address.

If you look at something like this it will show you what addresses a given host is associated with.

You can then try telneting to port 443 on each address to see if you get a connection or a connection refused on each one.