Automated Setup of Kibana and Elasticsearch with Filebeat Module in Elastic Cloud for Kubernetes (ECK)

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. Jamie Jackson wants to to this question. 。

I'm trying out the K8s Operator (aka ECK) and so far, so good.

However, I'm wondering what the right pattern is for, say, configuring Kibana and Elasticsearch with the Apache module.

I know I can do it ad hoc with:

filebeat setup --modules apache2 --strict.perms=false \
  --dashboards --pipelines --template \
  -E setup.kibana.host="${KIBANA_URL}"

But what's the automated way to do it? I see some docs for the Kibana dashboard portion of it but what about the rest (pipelines, etc.)?

Note: At some point, I may end up actually running a beat for the K8s cluster, but I'm not at that stage yet. At the moment, I just want to set Elasticsearch/Kibana up with the Apache module additions so that external Apache services' Filebeats can get ingested/displayed properly.

FYI, I'm on version 6.8 of the Elastic stack for now.

Answer 1

you can try auto-discovery using label based approach.

config:
filebeat.autodiscover:
 providers:
  - type: kubernetes
    hints.default_config.enabled: "false"
    templates:
      - condition.contains:
        kubernetes.labels.app: "apache"
    config:
    - module: apache
      access:
        enabled: true
        var.paths: ["/path/to/log/apache/access.log*"]
      error:
        enabled: true
        var.paths: ["/path/to/log/apache/error.log*"]