將帶有 filebeat 的日志發送到 logstash

Question

我正在嘗試讓 filebeat 將日志發送到另一台機器上的 logstash，但我無法讓它工作。 這是 filebeat.yml 配置：

https://pastebin.com/8a2RtGBa （由於字符限制使用pastebin）

這是具有logstash的機器上的配置：

[root@ELK ~]$ cat /etc/logstash/conf.d/testing.conf 
input{
file{
path => "/var/log/commands.log"
}
beats{
port => 5044
}
}
filter {
if [path] == "/var/log/commands.log" {
grok{
match => { "message" => "\[(%{TIMESTAMP_ISO8601:sys_timestamp})\]\s(?<field1>[0-9a-zA-Z_-]+)\s(?<field2>[0-9a-zA-Z_-]+)\:USER=(?<field3>[0-9a-zA-Z_-]+)\sPWD=(?<field4>[0-9a-zA-Z_/-]+)\sPID=\[(?<field5>[0-9]+)\]\sCMD=\"(?<field6>.*)\"\sExit=\[(?<field7>[0-9]+)\]\sCONNECTION=(?<field8>.*)"
}
}
}
}
output{
elasticsearch { hosts => ["localhost:9200"]}
}

如果我嘗試使用“filebeat setup -e”設置 filebeat，這就是我得到的：

[root@server150 ~]# filebeat setup -e
2020-07-17T08:12:36.199Z        INFO    instance/beat.go:647    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-17T08:12:36.200Z        INFO    instance/beat.go:655    Beat ID: aa84fd5b-d016-4688-a4a1-172dbcf2054a
2020-07-17T08:12:36.202Z        INFO    [beat]  instance/beat.go:983    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "aa84fd5b-d016-4688-a4a1-172dbcf2054a"}}}
2020-07-17T08:12:36.203Z        INFO    [beat]  instance/beat.go:992    Build info      {"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:15:37.000Z", "version": "7.8.0"}}}
2020-07-17T08:12:36.203Z        INFO    [beat]  instance/beat.go:995    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.10"}}}
2020-07-17T08:12:36.204Z        INFO    [beat]  instance/beat.go:999    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-09T11:44:16Z","containerized":false,"name":"server150","ip":["127.0.0.1/8","*ip-of-thismachine*/25"],"kernel_version":"3.10.0-1062.18.1.el7.x86_64","mac":["00:0c:29:e5:a1:fa"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"3eec437c66d444a59ef5f075a429441d"}}}
2020-07-17T08:12:36.204Z        INFO    [beat]  instance/beat.go:1028   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 12559, "ppid": 11800, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-07-17T08:12:35.340Z"}}}
2020-07-17T08:12:36.204Z        INFO    instance/beat.go:310    Setup Beat: filebeat; Version: 7.8.0
2020-07-17T08:12:36.205Z        INFO    [publisher]     pipeline/module.go:113  Beat name: server150
2020-07-17T08:12:36.206Z        WARN    beater/filebeat.go:156  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-07-17T08:12:36.206Z        ERROR   instance/beat.go:958    Exiting: Index management requested but the Elasticsearch output is not configured/enabled
Exiting: Index management requested but the Elasticsearch output is not configured/enabled

這是我嘗試使用“filebeat -e”運行 filebeat 時得到的結果：

[root@server150 ~]# filebeat -e
2020-07-17T08:16:47.104Z        INFO    instance/beat.go:647    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-17T08:16:47.104Z        INFO    instance/beat.go:655    Beat ID: aa84fd5b-d016-4688-a4a1-172dbcf2054a
2020-07-17T08:16:47.107Z        INFO    instance/beat.go:404    filebeat stopped.
2020-07-17T08:16:47.107Z        ERROR   instance/beat.go:958    Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

有誰知道發生了什么？ 我只是無法讓它工作。 非常感謝！

Answer 1

我向您推薦我的配置。

進程：*.log -> filebeat -> logstash -> elasticsearch

文件節拍.yml：

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/mylog/*.log

output.logstash:
  hosts: ["logstash:5044"]

logstash.conf：

input {
  beats {
    port => 5044
  }
}

filter {
  json {
    source => "message"
  }
  date{
    match => ["timestamp", "UNIX_MS"]
    target => "@timestamp"
  }
  ruby {
    code => "event.set('indexDay', event.get('[@timestamp]').time.localtime('+00:00').strftime('%Y%m%d'))"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    template => "/usr/share/logstash/templates/logstash.template.json"
    template_name => "logstash"
    template_overwrite => true
    index => "logstash-%{indexDay}"
    codec => json
  }
  stdout {
    codec => rubydebug
  }
}

logstash.template.json：

{
    "template": "logstash",
    "index_patterns": [
        "logstash-*"
    ],
    "settings": {
        "number_of_shards": 1
    },
    "mappings": {
        "properties": {
            "name": {
                "type": "keyword"
            },
            "class": {
                "type": "keyword"
            },
            "state": {
                "type": "integer"
            },
            "@timestamp": {
                "type": "date"
            }
        }
    }
}

您可以在下面的 github 存儲庫中找到一個簡單的解決方案。

docker-compose 配置： ELK with filebeat by docker-compose

Answer 2

go 到你的節拍的 yml 文件，並評論：

#setup.template.settings:
   #index.number_of_shards: 1

然后做：

filebeat setup -e

（不同節拍的相同工作）您最終將收到： Kibana dashboards successfully loaded. Loaded dashboards Kibana dashboards successfully loaded. Loaded dashboards