Unable to connect Filebeat to logstash for logging using ELK

Question

Hi I've been working on a automated logging using elastic stack. I have filebeat that is reading logs from the path and output is set to logstash over the port 5044 . The logstash config has an input listening to 5044 and output pushing to localhost:9200. The issue is I can't get it to work, I have no idea what's happening. Below are the files:

My filebeat.yml path: etc/filebeat/filebeat.yml

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/vol1/autosuggest/logs/*.log
#- c:\programdata\elasticsearch\logs\*
<other commented stuff>
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.10.XX.XX:5044"]

# Optional SSL. By default is off.
<other commented stuff>

---

My logstash.yml path: etc/logstash/logstash.yml

<other commented stuff>
path.data: /var/lib/logstash
<other commented stuff>
path.config: /etc/logstash/conf.d
<other commented stuff>

# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
http.host: "10.10.XX.XX"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
<other commented stuff>
path.logs: /var/log/logstash
<other commented stuff>

My logpipeline30aug.config file path: /usr/share/logstash

input {
  beats {
  port => 5044
  }
}

filter {
  grok {
match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:var0}%{SPACE}%{NOTSPACE}%{SPACE}(?<searchinfo>[^#]*)#(?<username>[^#]*)#(?<searchQuery>[^#]*)#(?<latitude>[^#]*)#(?<longitude>[^#]*)#(?<client_ip>[^#]*)#(?<responseTime>[^#]*)" }
  }
}

output {
   elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "logstash30aug2017"
      document_type => "log"
   }
}

Please Note: Elasticsearch, logstash, filebeat are all installed on the same machine with ip: 10.10.XX.XX and I've checked the firewall, it's not the issue for sure.

I checked that logstash, filebeat services are all running. Filebeat is able to push the data to elasticsearch when configured so and logstash is able to push the data to elasticsearch when configured so.

---

Maybe it's how I am executing the process is the issue.. I do a bin/logstash -f logpipeline30aug.config in /usr/share/logstash to start it and then I do a /etc/init.d/filebeat start from the root directory.

Please Note: Formatting may be effected due to stackoverflow formatting issue

Can someone please help? I've been trying everything since 3 days now, I've gone through the documentations as well

Answer 1

Your filebeat.yml looks invalid.

The output section lacks an indentation:

output.logstash:
  hosts: ["10.10.XX.XX:5044"]

In general, check the correctness of the config files to ensure they're ok.

For instance, for filebeat, you can run:

filebeat -c /etc/filebeat/filebeat.yml -configtest

If you have any errors it explains what is that error so you can fix it.

You can use a similar approach for other ELK services as well