使用 Elastic Common Schema 配置基于 Filebeat 提示的自动发现

Question

我找不到任何关于如何配置 filebeat 以处理 ECS 格式的 JSON 日志的文档。

我将ecs-pino-format用于 output“ECS”日志，这是一个典型的日志 I output：

{"log":{"level":"debug","logger":"pino"},"@timestamp":"2020-06-10T17:02:11.266Z","module":"APM","ecs":{"version":"1.5.0"},"message":"ended transaction {\"trans\":\"7614bf8a4895a7a4\",\"trace\":\"8a5c71d2c1c63f6dfc1a5bfd046701ed\",\"type\":\"request\",\"result\":\"HTTP 2xx\",\"name\":\"GET /healthcheck\"}"}

这是我的 filebeat 配置：

filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

使用此配置，我的日志不会被 Kibana 解释：

我将这个注释添加到我的 pod（甚至不确定我必须这样做......）：

co.elastic.logs/json.keys_under_root: true

这是我在 filebeat 上遇到的错误：

2020-06-10T16:47:00.773Z    WARN    [elasticsearch]    elasticsearch/client.go:384    Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x304e23a, ext:63727404418, loc:(*time.Location)(nil)}, Meta:null, Fields:{"agent":{"ephemeral_id":"cc9f9def-5d67-4592-8459-f556f8f2fc29","hostname":"filebeat-filebeat-4dqpq","id":"e8d9cffe-ceca-49f5-ae31-65bbb29353e8","type":"filebeat","version":"7.7.0"},"ecs":{"version":"1.5.0"},"host":{"name":"filebeat-filebeat-4dqpq"},"input":{"type":"container"},"json":{"@timestamp":"2020-06-10T16:46:58.049Z","ecs":{"version":"1.5.0"},"log":"","message":"sending span {\"span\":\"87ad75b7f0858817\",\"parent\":\"82e1f82870aa3e55\",\"trace\":\"13c7569f7562a72bef1300097d1ab86c\",\"name\":\"SELECT\",\"type\":\"db\"}","module":"APM","trace.id":"13c7569f7562a72bef1300097d1ab86c","transaction.id":"82e1f82870aa3e55"},"kubernetes":{"container":{"image":"registry.gitlab.com/consensys/codefi/products/assets/workflow-api:v0.1.3-2-g358bbc6","name":"generic-app"},"labels":{"app_kubernetes_io/instance":"workflow-api","app_kubernetes_io/name":"workflow-api","pod-template-hash":"b946b7c49"},"namespace":"codefi","node":{"name":"ip-192-168-33-94.eu-west-3.compute.internal"},"pod":{"name":"workflow-api-b946b7c49-7qldb","uid":"e984519d-8cc5-426d-bdac-e3f0dfa55c0b"},"replicaset":{"name":"workflow-api-b946b7c49"}},"log":{"file":{"path":"/var/log/containers/workflow-api-b946b7c49-7qldb_codefi_generic-app-9bff78b56f893e056e1e614de3c28aa6671dd4723c0dfc166460ac9bde43571a.log"},"offset":2303955},"stream":"stdout"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000ac8a90), Source:"/var/log/containers/workflow-api-b946b7c49-7qldb_codefi_generic-app-9bff78b56f893e056e1e614de3c28aa6671dd4723c0dfc166460ac9bde43571a.log", Offset:2304478, Timestamp:time.Time{wall:0xbfb060a48062556d, ext:986606661848, loc:(*time.Location)(0x3bdbf40)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x601c938, Device:0x10301}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [json.log] tried to parse field [log] as object, but found a concrete value"

如果我删除注释，我看不到任何错误，所以我猜它来自这个配置。

我在这里错过了一些文档吗？ 谢谢你的帮助。

Answer 1

找到了。 为了将来参考，当使用 ECS 日志格式和基于提示的自动发现时，只需将这些注释添加到您的 pod：

co.elastic.logs/json.keys_under_root: true
co.elastic.logs/json.message_key: message
co.elastic.logs/json.overwrite_keys: true

我希望这可以帮助别人！

Answer 2

我在 2022 年遇到了同样的问题，上面的注释不起作用，我找到了一篇关于使用 docker 登录弹性的博客，这启发了我，这就是我所做的：

filebeat.autodiscover:
 providers:
   - type: kubernetes
     node: ${NODE_NAME}
     hints.enabled: true
     hints.default_config:
       type: container
       paths:
         - /var/log/containers/*${data.kubernetes.container.id}.log


processors:
  # - add_cloud_metadata:
  # - add_host_metadata:
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 3
      target: ""
      overwrite_keys: true
      add_error_key: true
      expand_keys: true
#...

我在上面添加了处理器以从“消息”字段中提取字段。 希望它会帮助别人！

Answer 3

我使用了以下配置提供程序：- hints.default_config: paths: - '/var/log/containers/*-${data.container.id}.log' type: container hints.enabled: true host: '${HOSTNAME }'

和注释 co.elastic.logs/fileset= access

获取所有容器/pods 日志以在弹性搜索中查看