Push data directly from Filebeats to AWS ES managed service

Question

My issue is that I am trying to stream data from Filebeat to AWS ElasticSearch.

I approached this by providing the AWS endpoint in the beats output entry.

I tried both port 80 and 443 to no avail.

I checked this post, and from this I suppose that it is possible to push directly to AWSbut still cannot figure out how.

It would be really helpful if any of you has been through this and could shed some light!

Thank you!

Answer 1

Turns out it was a problem with permissions.

Make sure that the logs filebeat is trying to stream have the same permission as the filebeat.yml .

You can simply issue a chmod 777 to both files.

Finally, make sure, to prepend :443 after AWS ES endpoint.

Answer 2

I was using 7.10 version of Filebeat and Logstash.

Below blog

help me lot.

Steps are as:

Open filebeat.yml in any editor of your choice from location

/etc/filebeat/ on Linux or

C:\Program Files\filebeat-7.10.0 on windows

filebeat:
inputs:
– paths:
– E:/nginx-1.20.1/logs/.log
input_type: log


filebeat.config.modules:
enabled: true
path: ${path.config}/modules.d/*.yml


output:
logstash:
hosts: [“localhost:5044”]

Logstash Configuration

input {
beats {
port => 5044
ssl => false
}
}

filter {
grok {
match => [ “message” , “%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}”]
overwrite => [ “message” ]
}
mutate {
convert => [“response”, “integer”]
convert => [“bytes”, “integer”]
convert => [“responsetime”, “float”]
}
geoip {
source => “clientip”
target => “geoip”
add_tag => [ “nginx-geoip” ]
}
date {
match => [ “timestamp” , “dd/MMM/YYYY:HH:mm:ss Z” ]
remove_field => [ “timestamp” ]
}
useragent {
source => “agent”
}
}

output {
elasticsearch {
hosts => [“https://arun-learningsubway-ybalglooophuhyjmik3zmkmiq4.ap-south-1.es.amazonaws.com:443”]
index => “arun_nginx”
document_type => “%{[@metadata][type]}”
user => “myusername”
password => “mypassword”
manage_template => false
template_overwrite => false
ilm_enabled => false
}
}