[英]Filebeat > Logstash > ElasticSearch - Lumberjack Error
[英]error indexing into elasticsearch from filebeat and logstash
我建立了一个麋鹿堆栈,以在本地使用日志文件。 现在我正在尝试添加文件拍子,该文件将在索引到elasticsearch之前输出到logstash进行过滤。 这是我的配置文件beat.yml:
prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
paths:
- /var/samplelogs/wwwlogs/framework*.log
input_type: log
document_type: framework
logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
logging:
to_syslog: true
这是logstash配置:
input {
beats {
port => 5044
}
}
filter {
if [type] == "framework" {
grok {
patterns_dir => "/etc/logstash/conf.d/patterns"
match => {'message' => "\[%{WR_DATE:logtime}\] \[error\] \[app %{WORD:application}\] \[client %{IP:client}\] \[host %{HOSTNAME:host}\] \[uri %{URIPATH:resource}\] %{GREEDYDATA:error_message}"}
}
date {
locale => "en"
match => [ "logtime", "EEE MMM dd HH:mm:ss yyyy" ]
}
}
}
output {
elasticsearch {
host => "localhost"
port => "9200"
protocol => "http"
# manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
当我使用--configtest时,此logstash配置签出可以。 filebeat可以启动,但是在logstash.log中出现以下错误:
{:timestamp=>"2016-03-09T12:26:58.976000-0700", :message=>["INFLIGHT_EVENTS_REPORT", "2016-03-09T12:26:58-07:00", {"input_to_filter"=>20, "filter_to_output"=>20, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2016-03-09T12:27:03.977000-0700", :message=>["INFLIGHT_EVENTS_REPORT", "2016-03-09T12:27:03-07:00", {"input_to_filter"=>20, "filter_to_output"=>20, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2016-03-09T12:27:08.060000-0700", :message=>"Got error to send bulk of actions: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];", :level=>:error}
{:timestamp=>"2016-03-09T12:27:08.060000-0700", :message=>"Failed to flush outgoing items", :outgoing_count=>1, :exception=>"Java::OrgElasticsearchClusterBlock::ClusterBlockException", :backtrace=>["org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(org/elasticsearch/cluster/block/ClusterBlocks.java:151)", "org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(org/elasticsearch/cluster/block/ClusterBlocks.java:141)", "org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(org/elasticsearch/action/bulk/TransportBulkAction.java:215)", "org.elasticsearch.action.bulk.TransportBulkAction.access$000(org/elasticsearch/action/bulk/TransportBulkAction.java:67)", "org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(org/elasticsearch/action/bulk/TransportBulkAction.java:153)", "org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(org/elasticsearch/action/support/TransportAction.java:137)", "java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1142)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:617)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:warn}
{:timestamp=>"2016-03-09T12:27:08.977000-0700", :message=>["INFLIGHT_EVENTS_REPORT", "2016-03-09T12:27:08-07:00", {"input_to_filter"=>20, "filter_to_output"=>20, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2016-03-09T12:27:13.977000-0700", :message=>["INFLIGHT_EVENTS_REPORT", "2016-03-09T12:27:13-07:00", {"input_to_filter"=>20, "filter_to_output"=>20, "outputs"=>[]}], :level=>:warn}
这些错误不断重复出现。
在elasticsearch日志中,有一个错误invalidargumentexception:空文本。 我尝试将Logstash输出配置中的协议更改为“节点”。
在我看来,elasticsearch无法达到但正在运行:
$ curl localhost:9200
{
"status" : 200,
"name" : "Thena",
"version" : {
"number" : "1.1.2",
"build_hash" : "e511f7b28b77c4d99175905fac65bffbf4c80cf7",
"build_timestamp" : "2014-05-22T12:27:39Z",
"build_snapshot" : false,
"lucene_version" : "4.7"
},
"tagline" : "You Know, for Search"
}
这是我第一次尝试使用logstash。 谁能指出我正确的方向?
我能够使我的堆栈工作。 每个人的评论都是正确的,但在这种情况下,它恰好是我尚未完全理解的配置调整。
在log stash输出配置中的elasticsearch {}选项中,我注释掉了端口和协议(设置为9200和HTTP),并且可以正常工作。 我的第一个修复尝试是删除协议选项,因此默认情况下使用节点协议。 当那行不通时,我还删除了协议选项。 协议的默认值为“节点”,因此看来我根本无法使其通过HTTP工作,而我忘记了删除端口选项。 删除两者后,它起作用了。
这可能不会对将来的人们有所帮助,但是如果您要使用节点协议,请确保不要忘记从配置中删除端口选项-至少我认为这是我遇到的问题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.