如何从 logstash 连接 logstash？

Question

I want to use one logstash send data to multiple logstash services.

If I use this setting in k8s, it can work.

service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash
  name: logstash
spec:
  selector:
    app: logstash
  ports:
    - name: "5044"
      port: 5044
      targetPort: 5044

filebeat.yml

    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/log1.json
        tags: ["log1"]

    processors:
      - decode_json_fields:
          fields: ["message"]
          process_array: true
          max_depth: 1
          target: ""
          overwrite_keys: true
          add_error_key: false

    output.logstash:
      hosts: ["logstash-log1.default.svc.cluster.local:5044"]

logstash.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-logstash
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    xpack.monitoring.enabled: false
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }

    filter {
    }

    output {
      stdout { codec => rubydebug }
    }

But if I change to another way, set output from another logstash instance as:

k8s services:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash-log1
  name: logstash-log1
spec:
  selector:
    app: logstash-log1
  ports:
    - name: "5044"
      port: 5044
      targetPort: 5044

logstash.yml

    output {
      if "log1" in [tags] {
        stdout { codec => rubydebug }
        tcp {
          mode => "client"
          host => "logstash-log1.default.svc.cluster.local"
          port => 5044
          codec => "json"
        }
      }
    }

target logstash's logstash.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-logstash
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    xpack.monitoring.enabled: false
  logstash.conf: |
    input {
      tcp {
        port => 5044
      }
    }

    filter {
    }

    output {
      stdout { codec => rubydebug }
    }

The communication can be confimed, it works. But the data seems didn't send to logstash-log1.default.svc.cluster.local:5044 .

If use curl to test from origin logstash,

curl logstash-log1.default.svc.cluster.local:5044

Can get log data in the target logstash:

{
  "port" => 37424,
  "host" => "10.12.8.213"
}
{
  "port" => 37424,
  "host" => "10.12.8.213"
}
{
  "port" => 37424,
  "host" => "10.12.8.213"
}
{
  "port" => 37424,
  "host" => "10.12.8.213"
}

So the problem may be the configuration between 2 logstash. Doesn't tcp input/output methods work?

Answer 1

Try using the lumberjack output and beats input .

This is the recommended approach for Logstash-to-Logstash communication, and is explained in detail here in the logstash docs.

Answer 2

My solution thus far has been to use gelf between the two logstashes:

logstash-01: taking input from beats and sending it to logstash-02

/etc/logstash/conf.d/logstash-01.conf

input {
  beats {
    port => 5044
    }
}
output {
  gelf {
    id => "%{host}"
    host => "logstash-02.example.com"
    port => 9899
  }
}

logstash-02: receiving from logstash-01

/etc/logstash/conf.d/logstash-02.conf

input {
  gelf {
    host => "0.0.0.0"
    use_udp => true
    port => 9899
    strip_leading_underscore => false
    remap => false
  }
}

YMMV