如何在基于Linux的VM上运行的filebeat和在kubernetes中运行的logstash之间建立连接（logstash通过入口公开）

Question

I want to establish connectivity between filebeat running on a linux based VM and logstash running in kubernetes(logstash exposed through ingress). I have specified logstash output in the filebeat.yml file host: [" https://example.com/logstash "]. I have multiple services exposed through ingress and hence I want the logstash also to be exposed through the same ingress resource. The host attached to ingress resource is https://www.example.com . And I am using the .crt file of the TLS/SSL certificate associated with example.com in the certificate autority key of the filebeat.yml to secure the connection between filebeat and logstash. And in the logstash.yml which is running on kubernetes I have specified the input as TCP type instead of BEATS type.

Logstash service is running of ClusterIP Type and has the following ingress rule

   http:
     paths:
     - backend:
         serviceName: logstash-service
         servicePort: 5044
       path: /logstash

Now when I run this entire setup the filebeat on the VM gives me error

  output.go:100#011Failed to connect to backoff(async(tcp://https://example.com/logstash)): lookup https on 168.63.129.16:53: no such host
Jul 29 19:35:16 filebeat[75346]: 2019-07-29T19:35:16.954Z#011INFO#011pipeline/output.go:93#011Attempting to reconnect to backoff(async(tcp://https://example.com/logstash)) with 7 reconnect attempt(s)
Jul 29 19:35:16 filebeat[75346]: 2019-07-29T19:35:16.954Z#011DEBUG#011[logstash]#011logstash/async.go:111#011connect
Jul 29 19:35:16 filebeat[75346]: 2019-07-29T19:35:16.957Z#011WARN#011transport/tcp.go:53#011DNS lookup failure "https": lookup https on 168.63.129.16:53: no such host

Is this setup technically possible and sane? What should be the host what I should specify in filebeat and what should be the logstash input type? Will specifying the tls.crt specified in filebeat help in authenticating with /logstash?

Answer 1

It looks like you want to use a self-signed SSL certificate with a invalid hostname. There are two ways to make that hostname works in your self managed network:

• add a IP record in /etc/hosts for the hostname, the hostname https://example.com/logstash will work in your filebeat config.
• create the certificate and add a SubjectAltName (SAN) of your IP, change your filebeat config, use IP instead of https://example.com/logstash hostname.