Filebeat 7.2-將日志從Docker容器保存到Logstash

Question

我的ec2實例上運行了幾個Docker容器。

我想將這些容器中的日志直接保存到Logstash（彈性雲）中。

當我嘗試手動安裝Filebeat時，一切正常。 我已經使用下載了它

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.2.0-linux-x86_64.tar.gz

我已將其解壓縮，將filebeat.yml配置更改為

filebeat.inputs:

- type: log

  enabled: true

  fields:
    application: "myapp"

  fields_under_root: true

  paths:
    - /var/lib/docker/containers/*/*.log

cloud.id: "iamnotshowingyoumycloudidthisisjustfake"
cloud.auth: "elastic:mypassword"

這工作得很好，我可以在Kibana中搜索應用程序“ myapp”后找到我的日志。

但是，當我嘗試從Docker運行Filebeat時，沒有成功。

這是我的docker-compose.yml的filebeat部分

filebeat:
    image: docker.elastic.co/beats/filebeat:7.2.0
    container_name: filebeat
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock #needed for autodiscover

我以前的手動執行filebeat.yml文件不起作用，因此我嘗試了許多示例，但沒有任何效果。 這是一個我認為應該起作用的示例，但事實並非如此。 Docker容器啟動沒有問題，但是無法以某種方式從日志文件讀取。

filebeat.autodiscover:
  providers:
    - type: docker

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  json.keys_under_root: true
  json.add_error_key: true
  fields_under_root: true
  fields:
    application: "myapp"

cloud.id: "iamnotshowingyoumycloudidthisisjustfake"
cloud.auth: "elastic:mypassword"

我也嘗試過這樣的事情

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        config:
          - type: docker
            containers.ids:
              - "*"

filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
    processors:
      - add_docker_metadata:
    fields:
      application: "myapp"
    fields_under_root: true

cloud.id: "iamnotshowingyoumycloudidthisisjustfake"
cloud.auth: "elastic:mypassword"

我不知道還有什么嘗試，filebeat日志仍然顯示

 "harvester":{"open_files":0,"running":0}}

我100％確保來自容器的日志位於/var/lib/docker/containers/*/*.log ...下，正如我所說的，當手動安裝Filebeat時，它可以工作，而不是作為docker映像。

有什么建議嗎？

Filebeat的輸出日志

2019-07-23T05:35:58.128Z        INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.2.0
2019-07-23T05:35:58.128Z        INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'filebeat-7.2.0' as ILM is enabled.
2019-07-23T05:35:58.129Z        INFO    elasticsearch/client.go:166     Elasticsearch url: https://123456789.us-east-1.aws.found.io:443
2019-07-23T05:35:58.129Z        INFO    [publisher]     pipeline/module.go:97   Beat name: e3e5163f622d
2019-07-23T05:35:58.136Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-07-23T05:35:58.142Z        INFO    instance/beat.go:421    filebeat start running.
2019-07-23T05:35:58.142Z        INFO    registrar/migrate.go:104        No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2019-07-23T05:35:58.142Z        INFO    registrar/migrate.go:112        Initialize registry meta file
2019-07-23T05:35:58.144Z        INFO    registrar/registrar.go:108      No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2019-07-23T05:35:58.146Z        INFO    registrar/registrar.go:145      Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2019-07-23T05:35:58.146Z        INFO    registrar/registrar.go:152      States Loaded from registrar: 0
2019-07-23T05:35:58.146Z        INFO    crawler/crawler.go:72   Loading Inputs: 1
2019-07-23T05:35:58.146Z        WARN    [cfgwarn]       docker/input.go:49      DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2019-07-23T05:35:58.150Z        INFO    log/input.go:148        Configured paths: [/var/lib/docker/containers/*/*.log]
2019-07-23T05:35:58.150Z        INFO    input/input.go:114      Starting input of type: docker; ID: 11882227825887812171
2019-07-23T05:35:58.150Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2019-07-23T05:35:58.150Z        WARN    [cfgwarn]       docker/docker.go:57     BETA: The docker autodiscover is beta
2019-07-23T05:35:58.153Z        INFO    [autodiscover]  autodiscover/autodiscover.go:105        Starting autodiscover manager
2019-07-23T05:36:28.144Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        
{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":17}},"total":{"ticks":40,"time":{"ms":52},"value":40},"user":{"ticks":30,"time":{"ms":35}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":9},"info":{"ephemeral_id":"4427db93-2943-4a8d-8c55-6a2e64f19555","uptime":{"ms":30111}},"memstats":{"gc_next":4194304,"memory_alloc":2118672,"memory_total":6463872,"rss":28352512},"runtime":{"goroutines":34}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":1},"load":{"1":0.31,"15":0.03,"5":0.09,"norm":{"1":0.31,"15":0.03,"5":0.09}}}}}}

Answer 1

嗯，在Filebeat配置中我看不到任何明顯的原因，為什么它不起作用，我有一個非常相似的配置正在運行6.x Filebeat。

我建議對容器進行docker inspect ，並確認安裝在那里，也許檢查權限，但日志中可能會顯示錯誤。

還可以嘗試使用container input嗎？ 我相信這是在7.2+版本中推薦的容器日志方法： https ://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-input-container.html