如何使用 Logstash 解析来自 S3 的数据并推送到 Elastic Search，然后推送到 Kibana

Question

I have a log file created in S3 bucket every minute. The data is "\\x01" delimited. One of the column is a timestamp field.

I want to load this data into elastic search.

I tried using the following logstash conf. But it doesn't seem to work. I don't see any output. I took some reference from http://brewhouse.io/blog/2014/11/04/big-data-with-elk-stack.html

Logstash config file is as follows:

input {
  s3 {
    bucket => "mybucketname"
    credentials => [ "accesskey", "secretkey" ]
  }
}
filter {
  csv {
   columns => [ "col1", "col2", "@timestamp" ]
   separator => "\x01"
  }
}
output {
  stdout { } 
}

How do I modify this file to take in new file coming in every minute?

I would then eventually want to connect Kibana to ES to visualize the changes.

Answer 1

Just use logstash-forwarder to send the files from S3, you will have to generate certificates for authorization.

There is a really nice tutorial: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-logs-on-centos-7

if you getting I/O errors, mb you can solve them by setting cluster:

inside logstash.conf:

output {
    elasticsearch {
        host => "127.0.0.1"
        cluster => CLUSTER_NAME
    }

inside elasticsearch.yml:

cluster.name: CLUSTER_NAME

if you getting problems generating certificates, you can generate them using this: https://raw.githubusercontent.com/driskell/log-courier/develop/src/lc-tlscert/lc-tlscert.go

I also found better init.d for logstash-forwarder on CentOS: http://smuth.me/posts/centos-6-logstash-forwarder-init-script.html