Filebeat 監控指標在 ElasticSearch 中不可見

Question

我正在使用 Filebeat 從 PubSub 攝取事件並將它們發送到 ES 以進行索引/可視化。 我注意到在特定的高測試負載下——並非所有事件都到達 ES。 所以我正在嘗試調試管道 - 試圖找出下降發生的位置。

我希望通過監控 Filebeat 本身並將指標發送到同一個 ES 集群（托管在 elastic.io 上），我可以深入了解 Filebeat 中發生的事情。

所以我做了：

-- 在 Elastic.io 集群中啟用 XPack 監控如下：

-- 在 filebeat.yaml 中啟用監控：

monitoring.enabled: true
monitoring.elasticsearch:
   
  api_key: ${ES_API_KEY}

彈性 output 配置如下：

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: true
  index: "ibc-parsed-logs"
  parameters.pipeline: "geoip-info"
  hosts: ${ES_HOSTS}

  # Authentication credentials - either API key or username/password.
  api_key: ${ES_API_KEY}

根據 Elastic 文檔 - 如果我使用 elasticsearch output - 那么集群 ID/auth/credentials 將從上面的 output 配置中確定...

我還啟用了監控指標的日志記錄：

logging.metrics.enabled: true

當我使用此配置運行 Filebeat 時，我看到確實收集了監控指標 - 我看到很多日志，例如：

2022-09-30T01:58:49.765Z    INFO    [monitoring]    log/log.go:192  Total metrics   {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"/","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"/","total":{"ns":1609969280422}},"memory":{"id":"/","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":59994112}}}},"cpu":{"system":{"ticks":950350,"time":{"ms":950351}},"total":{"ticks":1608520,"time":{"ms":1608525},"value":1608520},"user":{"ticks":658170,"time":{"ms":658174}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"2f0fb51b-0dc7-4ea6-97ea-d9f07f7a9dd6","uptime":{"ms":15354077},"version":"7.15.0"},"memstats":{"gc_next":27183328,"memory_alloc":25632752,"memory_sys":77874184,"memory_total":51893040880,"rss":132669440},"runtime":{"goroutines":19}},"filebeat":{"events":{"active":0,"added":3095135,"done":3095135},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"events":{"acked":3055775,"active":100,"batches":62013,"dropped":0,"duplicates":39360,"failed":0,"toomany":0,"total":3095235},"read":{"bytes":61600055,"errors":3},"type":"elasticsearch","write":{"bytes":3728037960,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":3095135,"retry":350,"total":3095135},"queue":{"acked":3095135,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":8},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}

但是，當我 go 到ES 集群 -> 可觀察性 -> 指標 -> 清單時，我只看到這條消息：“看起來你沒有任何指標索引。 ” - 並且沒有任何指標 - Kibana 中沒有任何內容，沒有索引任何指標...

為什么不向 ES 發送/顯示指標？ 我錯過了其他一些配置設置嗎？

謝謝！ 碼頭

更新 1：根據以下答案中的建議 - 我已為所有類別啟用 DEBUG 日志記錄 - 並且可以在 filebeat 日志中看到很多其他信息。 據我所見 - 見下面的摘錄 - Fielbeat 確實可以很好地連接到 ES 並將指標數據發送到 ES。 但是當我 go 到 ES - 任何指標數據都不存在索引。 我看到的唯一索引是我從 pubsub 收集的真實數據/事件的索引，以及一些與 APM 相關的索引。

完整日志： https://controlc.com/8021ff33

片段：

 2022-10-04T14:25:32.917Z DEBUG [esclientleg] transport/logging.go:41 Completed dialing successfully {.network": "tcp", "address": "XXX.us-east4.gcp.elastic-cloud.com:443"} 2022-10-04T14:25:32.945Z DEBUG [esclientleg] eslegclient/connection.go:272 Ping status code: 200 2022-10-04T14:25:32.946Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0 2022-10-04T14:25:32.947Z DEBUG [esclientleg] eslegclient/connection.go:328 GET https://XXX.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled <nil> 2022-10-04T14:25:32.982Z DEBUG [monitoring] elasticsearch/client.go:101 XPack monitoring is enabled 2022-10-04T14:25:32.983Z INFO [monitoring] elasticsearch/elasticsearch.go:244 Successfully connected to X-Pack Monitoring endpoint. 2022-10-04T14:25:32.984Z DEBUG [monitoring] elasticsearch/elasticsearch.go:250 Finish monitoring endpoint init loop. 2022-10-04T14:25:32.984Z INFO [monitoring] elasticsearch/elasticsearch.go:258 Start monitoring state metrics snapshot loop with period 1m0s. 2022-10-04T14:25:32.984Z INFO [monitoring] elasticsearch/elasticsearch.go:258 Start monitoring stats metrics snapshot loop with period 10s. 2022-10-04T14:25:41.061Z DEBUG [input] input/input.go:139 Run input 2022-10-04T14:25:42.959Z DEBUG [monitoring] processing/processors.go:203 Publish event: { "@timestamp": "2022-10-04T14:25:42.950Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "7.15.0", "type": "beats_stats", "interval_ms": 10000, "params": { "interval": "10s" } }, "beat": { "type": "filebeat", "version": "7.15.0", "name": "9975cbe98075", "host": "9975cbe98075", "uuid": "08e8a88e-e214-4d48-a65c-d5b5226776a5" }, "metrics": { "system": { "cpu": { "cores": 8 }, "load": { "1": 0.04, "5": 0.01, "15": 0, "norm": { "1": 0.005, "5": 0.0013, "15": 0 } } }, "beat": { "cgroup": { "cpuacct": { "id": "/", "total": { "ns": 596922278 } }, "memory": { "id": "/", "mem": { "limit": { "bytes": 9223372036854771712 }, "usage": { "bytes": 46735360 } } }, "cpu": { "stats": { "periods": 0, "throttled": { "ns": 0, "periods": 0 } }, "id": "/", "cfs": { "period": { "us": 100000 }, "quota": { "us": 0 } } } }, "handles": { "open": 20, "limit": { "hard": 1048576, "soft": 1048576 } }, "info": { "uptime": { "ms": 12033 }, "ephemeral_id": "3dac65ba-ee80-4333-8eeb-e46106b369c8", "version": "7.15.0" }, "memstats": { "memory_alloc": 13034112, "memory_sys": 76104712, "gc_next": 21276432, "rss": 116137984, "memory_total": 64822632 }, "cpu": { "total": { "time": { "ms": 549 }, "value": 540, "ticks": 540 }, "user": { "time": { "ms": 323 }, "ticks": 320 }, "system": { "ticks": 220, "time": { "ms": 226 } } }, "runtime": { "goroutines": 71 } }, "registrar": { "states": { "current": 0, "update": 0, "cleanup": 0 }, "writes": { "success": 0, "total": 0, "fail": 0 } }, "filebeat": { "harvester": { "started": 0, "closed": 0, "running": 0, "open_files": 0, "skipped": 0 }, "input": { .netflow": { "packets": { "dropped": 0, "received": 0 }, "flows": 0 }, "log": { "files": { "renamed": 0, "truncated": 0 } } }, "events": { "done": 0, "active": 0, "added": 0 } }, "libbeat": { "output": { "read": { "bytes": 0, "errors": 0 }, "type": "elasticsearch", "events": { "batches": 0, "total": 0, "acked": 0, "failed": 0, "dropped": 0, "duplicates": 0, "active": 0, "toomany": 0 }, "write": { "errors": 0, "bytes": 0 } }, "pipeline": { "clients": 1, "events": { "active": 0, "total": 0, "filtered": 0, "published": 0, "failed": 0, "dropped": 0, "retry": 0 }, "queue": { "acked": 0, "max_events": 4096 } }, "config": { "scans": 0, "reloads": 0, "module": { "starts": 0, "stops": 0, "running": 0 } } } } } 2022-10-04T14:25:42.964Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(monitoring(https://XXX.us-east4.gcp.elastic-cloud.com:443)) 2022-10-04T14:25:42.964Z DEBUG [monitoring] elasticsearch/client.go:66 Monitoring client: connect. 2022-10-04T14:25:42.965Z DEBUG [esclientleg] eslegclient/connection.go:249 ES Ping(url=https://XXX.us-east4.gcp.elastic-cloud.com:443) 2022-10-04T14:25:42.964Z INFO [monitoring] pipeline/retry.go:219 retryer: send unwait signal to consumer 2022-10-04T14:25:42.966Z INFO [monitoring] pipeline/retry.go:223 done 2022-10-04T14:25:43.015Z DEBUG [esclientleg] eslegclient/connection.go:272 Ping status code: 200 2022-10-04T14:25:43.016Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0 2022-10-04T14:25:43.017Z DEBUG [esclientleg] eslegclient/connection.go:328 GET https://XXX.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled <nil> 2022-10-04T14:25:43.205Z DEBUG [monitoring] elasticsearch/client.go:101 XPack monitoring is enabled 2022-10-04T14:25:43.207Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(monitoring(https://XXX.us-east4.gcp.elastic-cloud.com:443)) established 2022-10-04T14:25:43.239Z DEBUG [monitoring] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 1]

更新 2：10/06/2022：在進一步努力獲取實際的 ES 日志之后（這在 elastic.io 的托管 ES 集群上並不那么簡單：！）- 我終於能夠訪問它們並注意到以下許多警告:

16:32:50.998
elasticsearch.server
[elasticsearch.server][WARN] Authentication to realm found failed - Password authentication failed for ec-local-beats-monitor

我不確定這個用戶可能是誰：“ec-local-beats-monitor”？？ 以及它的配置/設置位置——絕對不在我的 filebeat.yml 配置中，用於 Filebeat 進程……

這可能與丟失/未將 Filebeat 指標數據索引到 ES 中的問題有關嗎？

Answer 1

Tldr；

我認為您的帖子中有多個問題。

1. 如何調查丟失的消息？
2. 如何訪問filebeat的監控數據？

我可能錯過了它，但始終歡迎發布您正在運行的堆棧版本。

解決方案（也許）

1. 掉線的消息

這里有一些想法。

• 你可能有一個掉落處理器嗎
• 檢查 filebeat 日志以了解處理器上的錯誤
• 檢查攝取管道是否有錯誤？

2.監控

根據文檔，您應該在堆棧監控下找到它

Answer 2

嘗試更改日志級別以更深入地了解日志。

要獲取 filebeat 指標，Kibana -> Management -> Saved Objects 並刪除與 beats 相關的所有默認值並嘗試重新運行它。 大多數默認索引模板配置不正確，這可能導致指標在 kibana 上不可用。