How to configure RabbitMQ with Filebeat

Question

We use ELK to control our logs and visualize them in Kibana. We have about 30 different log types, that we are sending both from our grid machines and both from our client side machines.

The client side machines work with filebeat-->logstash-->elastic-->Kibana

I want to change the config from:
filebeat-->logstash-->elastic-->Kibana
to:
filebeat-->Rabbitmq-->logstash-->elastic-->Kibana

In elastic I founded this yml examples - when RabbitMQ isn't one of them .
Is it possible to config our system in this way? or I have to use:
filebeat-->logstash1-->Rabbitmq-->logstash2-->elastic-->Kibana

Answer 1

I'm not familiar with RabbitMQ in this context, but this seems to be an open issue in the Github repository for Beats:

https://github.com/elastic/beats/issues/581

There's also this post from the Elastic forums that indicates filebeat doesn't have plugin support, so I'm guessing the first option is no :

https://discuss.elastic.co/t/filebeat-rabbitmq-plugin/55189

Logstash certainly supports rabbitmq as an output:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-rabbitmq.html

Answer 2

I found in the end a nice implementation (although it's not an "official" one) for a FileBeat output plugin . Because as wrote here correctly Beats team doesn't going to support RabbitMQ plugin someone wrote it himself.

In Summery, in order to use the plugin you should do the following:

1. In the main.go file:
   package main

    import ( "os" _ "github.com/sidleal/mqttout" "github.com/sidleal/countbeat/cmd" ) func main() { if err := cmd.RootCmd.Execute(); err != nil { os.Exit(1) } } 

2. In the Config filebeat (yourbeat.yml):
   config file

    output.mqtt: host: "127.0.0.1" port: 1883 topic: "mytopic" user: "myvhost:myuser" password: "mypassword" 

3. bind amq.topic exchange to your desired queue, putting your topic in Routing Key.