堆棧內存溢出
  簡體   English   中英

Filebeat 不使用自動發現收集日志

[英]Filebeat not harvesting logs with autodiscover

 原文  2022-07-02 18:06:23  6   1    elasticsearch/ kubernetes-helm/ filebeat

問題描述

我在突然停止將日志發送到 elasticsearch 的環境中遇到 Filebeat 問題。 在這兩種環境中，我們都有相同的設置，但在這個環境中它剛剛停止。Filebeat、ElasticSearch 和 Kibana 版本 7.15.0 所有 helm 部署 /var/lib/docker/containers/ 在 filebeat 容器上都是空的，但在另一個容器中也是如此工作環境..

Filebeat 日志：

2022-07-02T16:56:12.731Z        DEBUG   [input] input/input.go:139      Run input
2022-07-02T16:56:12.731Z        DEBUG   [input] log/input.go:215        Start next scan {"input_id": "31e0e6d8-e599-453a-a8d0-69afdf5b52d6"}
2022-07-02T16:56:12.731Z        DEBUG   [input] log/input.go:279        input states cleaned up. Before: 0, After: 0, Pending: 0        {"input_id": "31e0e6d8-e599-453a-a8d0-69afdf5b52d6"}
2022-07-02T16:56:12.976Z        DEBUG   [input] input/input.go:139      Run input
2022-07-02T16:56:12.976Z        DEBUG   [input] log/input.go:215        Start next scan {"input_id": "89b55ab8-8fb3-49c4-9d9e-2372c956cf49"}
2022-07-02T16:56:12.977Z        DEBUG   [input] log/input.go:279        input states cleaned up. Before: 0, After: 0, Pending: 0        {"input_id": "89b55ab8-8fb3-49c4-9d9e-2372c956cf49"}
2022-07-02T16:56:13.074Z        DEBUG   [input] input/input.go:139      Run input
2022-07-02T16:56:13.074Z        DEBUG   [input] input/input.go:139      Run input
2022-07-02T16:56:13.074Z        DEBUG   [input] input/input.go:139      Run input
2022-07-02T16:56:13.074Z        DEBUG   [input] log/input.go:215        Start next scan {"input_id": "ac5b2c6d-189a-420a-bb00-f9d9e6d5aef7"}
2022-07-02T16:56:13.074Z        DEBUG   [input] log/input.go:215        Start next scan {"input_id": "be885467-72ea-44c1-bdce-cdd91fb03e79"}
2022-07-02T16:56:13.074Z        DEBUG   [input] log/input.go:215        Start next scan {"input_id": "1fa30d44-77e8-42ec-8d22-55abd4f8f60b"}
2022-07-02T16:56:13.074Z        DEBUG   [input] input/input.go:139      Run input
2022-07-02T16:56:13.074Z        DEBUG   [input] log/input.go:279        input states cleaned up. Before: 0, After: 0, Pending: 0        {"input_id": "ac5b2c6d-189a-420a-bb00-f9d9e6d5aef7"}
2022-07-02T16:56:13.074Z        DEBUG   [input] log/input.go:279        input states cleaned up. Before: 0, After: 0, Pending: 0        {"input_id": "1fa30d44-77e8-42ec-8d22-55abd4f8f60b"}
2022-07-02T16:56:13.074Z        DEBUG   [input] log/input.go:279        input states cleaned up. Before: 0, After: 0, Pending: 0        {"input_id": "be885467-72ea-44c1-bdce-cdd91fb03e79"}

在 filebeat 容器內：

ls data/registry/filebeat
log.json
meta.json


cat logs/filebeat
2022-07-02T17:37:30.639Z        INFO    instance/beat.go:665    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2022-07-02T17:37:30.640Z        DEBUG   [beat]  instance/beat.go:723    Beat metadata path: /usr/share/filebeat/data/meta.json
2022-07-02T17:37:30.640Z        INFO    instance/beat.go:673    Beat ID: b0e19db9-df61-4eec-9a95-1cd5ef653718
2022-07-02T17:37:30.640Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.15.0' as ILM is enabled.
2022-07-02T17:37:30.641Z        INFO    [esclientleg]   eslegclient/connection.go:100   elasticsearch url: http://elasticsearch.logging:9200
2022-07-02T17:37:30.740Z        DEBUG   [esclientleg]   eslegclient/connection.go:249   ES Ping(url=http://elasticsearch.logging:9200)
2022-07-02T17:37:30.742Z        DEBUG   [esclientleg]   transport/logging.go:41 Completed dialing successfully  {"network": "tcp", "address": "elasticsearch.logging:9200"}
2022-07-02T17:37:30.743Z        DEBUG   [esclientleg]   eslegclient/connection.go:272   Ping status code: 200
2022-07-02T17:37:30.743Z        INFO    [esclientleg]   eslegclient/connection.go:273   Attempting to connect to Elasticsearch version 7.15.0
2022-07-02T17:37:30.743Z        DEBUG   [esclientleg]   eslegclient/connection.go:328   GET http://elasticsearch.logging:9200/_license?human=false  <nil>

cat data/meta.json
{"uuid":"b0e19db9-df61-4eec-9a95-1cd5ef653718","first_start":"2022-05-29T00:10:26.137238912Z"}

ls data/registry/filebeat
log.json
meta.json

cat data/registry/filebeat/log.json

cat data/registry/filebeat/meta.json
{"version":"1"}

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 1e66a1c066aa10de73834586c605c7adf71b2c652498b0de7a9d94b44633f919
    cni.projectcalico.org/podIP: 10.0.4.120/32
    cni.projectcalico.org/podIPs: 10.0.4.120/32
    co.elastic.logs/enabled: "false"
    configChecksum: 9e8011c4cd9f9bf36cafe98af8e7862345164b1c11f062f4ab9a67492248076
    kubectl.kubernetes.io/restartedAt: "2022-04-14T16:22:07+03:00"
  creationTimestamp: "2022-07-01T13:53:29Z"
  generateName: filebeat-filebeat-
  labels:
    app: filebeat-filebeat
    chart: filebeat-7.15.0
    controller-revision-hash: 79bdd78b56
    heritage: Helm
    pod-template-generation: "21"
    release: filebeat
  name: filebeat-filebeat-95l2d
  namespace: logging
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: filebeat-filebeat
    uid: 343f6f76-ffde-11e9-bf3f-42010a9c01ac
  resourceVersion: "582889515"
  uid: 916d7dc9-f4b2-498a-9963-91213f568560
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - ..mynode
  containers:
  - args:
    - -e
    - -E
    - http.enabled=true
    env:
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: ELASTICSEARCH_HOSTS
      value: elasticsearch.logging:9200
    image: docker.elastic.co/beats/filebeat:7.15.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash -e
          curl --fail 127.0.0.1:5066
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: filebeat
    readinessProbe:
      exec:
        command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash -e
          filebeat test output
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      limits:
        cpu: 100m
        memory: 200Mi
      requests:
        cpu: 50m
        memory: 50Mi
    securityContext:
      privileged: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /usr/share/filebeat/filebeat.yml
      name: filebeat-config
      readOnly: true
      subPath: filebeat.yml
    - mountPath: /usr/share/filebeat/my_ilm_policy.json
      name: filebeat-config
      readOnly: true
      subPath: my_ilm_policy.json
    - mountPath: /usr/share/filebeat/data
      name: data
    - mountPath: /var/lib/docker/containers
      name: varlibdockercontainers
      readOnly: true
    - mountPath: /var/log
      name: varlog
      readOnly: true
    - mountPath: /var/run/docker.sock
      name: varrundockersock
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-2gvbn
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ..mynode
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: filebeat-filebeat
  serviceAccountName: filebeat-filebeat
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  volumes:
  - configMap:
      defaultMode: 384
      name: filebeat-filebeat-daemonset-config
    name: filebeat-config
  - hostPath:
      path: /var/lib/filebeat-filebeat-logging-data
      type: DirectoryOrCreate
    name: data
  - hostPath:
      path: /var/lib/docker/containers
      type: ""
    name: varlibdockercontainers
  - hostPath:
      path: /var/log
      type: ""
    name: varlog
  - hostPath:
      path: /var/run/docker.sock
      type: ""
    name: varrundockersock
  - name: kube-api-access-3axln
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

1 個解決方案

解決方案1
 0  2022-07-06 13:33:45

實際上，它適用於 elastic.co 網站上發布的另一個配置：

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*-${data.container.id}.log  # CRI path

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html

我仍然不確定為什么會突然發生這種情況，但原因可能是節點上 kubernetes 的容器運行時更​​改，但我無權檢查

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 即使收獲成功，Filebeat 也不會將日志發送到 Logstash Docker 文件節拍自動發現未檢測到 nginx 日志 FileBeat收集問題 Filebeat沒有將日志推送到Elasticsearch Filebeat未將日志發送到Logstash 使用FileBeat發送日志問題 將帶有 filebeat 的日志發送到 logstash Docker Filebeat Nginx 日志 使用 Elastic Common Schema 配置基於 Filebeat 提示的自動發現 Filebeat vs Rsyslog用於轉發日志
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM