如何在kubernetes的单个filebeat DaemonSet中声明多个output.logstash？

Question

I have 2 applications (Application1, Application2) running on the Kubernetes cluster. I would like to collect the logs from my applications from outside of the Kubernetes cluster and save them in different directories(for eg: /var/log/application1/application1-YYYYMMDD.log and /var/log/application2/application2-YYYYMMDD.log).

Therefore I deploy a filebeat DaemonSet on the Kubernetes cluster to fetch the logs from my applications(Application1, Application2) and run logstash service on the instance where I want to save the log files(outside of the Kubernetes cluster).

I create 2 filebeat.yml(filebeat-application1.yml and filebeat-application2.yml) files in configMap and then feed both files as args in DaemonSet(docker.elastic.co/beats/filebeat:7.10.1) as below.

....
  - name: filebeat-application1
    image: docker.elastic.co/beats/filebeat:7.10.1
    args: [
      "-c", "/etc/filebeat-application1.yml",
      "-c", "/etc/filebeat-application2.yml",
      "-e",
    ]
.....

But only /etc/filebeat-application2.yml is affected. Therefore, I get log only from application2.

Can you please help me about how to feed two filebeat configuration files into docker.elastic.co/beats/filebeat DaemonSet? or how to config two "filebeat.autodiscovery:" rules with 2 separate "output.logstash:"?

Below is my complete filebeat-kubernetes-whatsapp.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config-application1
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat-application1.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: default
            - condition:
                contains:
                  kubernetes.pod.name: "application1"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.pod.name}*.log
    processors:
      - add_locale:
          format: offset
      - add_kubernetes_metadata:
    output.logstash:
      hosts: ["IP:5045"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config-application2
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat-application2.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: default
            - condition:
                contains:
                  kubernetes.pod.name: "application2"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.pod.name}*.log
    processors:
      - add_locale:
          format: offset
      - add_kubernetes_metadata:
    output.logstash:
      hosts: ["IP:5044"]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat-application1
        image: docker.elastic.co/beats/filebeat:7.10.1
        args: [
          "-c", "/etc/filebeat-application1.yml",
          "-c", "/etc/filebeat-application2.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config-application1
          mountPath: /etc/filebeat-application1.yml
          readOnly: true
          subPath: filebeat-application1.yml
        - name: config-application2
          mountPath: /etc/filebeat-application2.yml
          readOnly: true
          subPath: filebeat-application2.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: config-application1
        configMap:
          defaultMode: 0640
          name: filebeat-config-application1
      - name: config-application2
        configMap:
          defaultMode: 0640
          name: filebeat-config-application2
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
---

Answer 1

It is not possible, filebeat supports only one output.

From the documentation

Only a single output may be defined.

You will need to send your logs to the same logstash instance and filter the output based on some field.

For example, assuming that you have the field kubernetes.pod.name in the event sent to logstash, you could use something like this.

output {
    if [kubernetes][pod][name] == "application1" {
        your output for the application1 log
    }
    if [kubernetes][pod][name] == "application2" {
        your output for the application2 log
    }
}

Answer 2

I found the working way for my problem. Maybe it is not the correct way but it can meet my requirement.

filebeat-kubernetes-whatsapp.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: default
            - condition:
                contains:
                  kubernetes.pod.name: "application1"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}*.log
            - condition:
                contains:
                  kubernetes.pod.name: "application2"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}*.log
    processors:
      - add_locale:
            format: offset
      - add_kubernetes_metadata:
            host: ${NODE_NAME} 
            matchers:
              - logs_path:
                  logs_path: "/var/log/containers/"
    output.logstash:
      hosts: ["IP:5044"]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.10.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
---

/etc/logstash/conf.d/config.conf

input {
  beats {
    port => 5044
  }
}
    
#filter {
#  ...
#}

output {
  if "application1" in [kubernetes][pod][name] {
    file {
      enable_metric => false
      gzip => false 
      codec => line { format => "[%{[@timestamp]}] [%{[kubernetes][node][name]}/%{[kubernetes][pod][name]}/%{[kubernetes][pod][uid]}] [%{message}]"}
      path => "/abc/def/logs/application1%{+YYYY-MM-dd}.log"
    }
  }
  if "application2" in [kubernetes][pod][name] {
    file {
      enable_metric => false
      gzip => false 
      codec => line { format => "[%{[@timestamp]}] [%{[kubernetes][node][name]}/%{[kubernetes][pod][name]}/%{[kubernetes][pod][uid]}] [%{message}]"}
      path => "/abc/def/logs/application2%{+YYYY-MM-dd}.log"
    }
  }
}