如何使用 filebeat 从 juniper 交换机获取日志消息以供 ELK 堆栈处理？

Question

I apologize in advanced as I am fairly new to using stack overflow. Please let me know if I need to change anything with my question.

I am currently trying to set up a log aggregator for my work. My plan is to use the elastic stack and filebeat to gather the logs and display them. I used this tutorial to setup the initial server:

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04

I was able to read the syslogs from the system using the filebeat system module just like in the tutorial. I then disabled it so that I could try to log the syslogs from a juniper switch without the interference of having to read through all the server syslogs.

Right now I am just trying to get the logs to come in from a juniper switch to my virtual machine. I have the switch set up to send syslogs to the virtual machine's ip on port 9513. I can see the logs coming in when I use tcpdump to monitor the port.

user@logserver:~$ sudo tcpdump -i ens160 port 9513
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
16:42:20.558768 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 124
16:45:00.571980 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 79
16:45:00.576370 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
16:47:45.944277 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 124
16:48:20.568840 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 64
16:50:00.653082 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
16:50:55.769884 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 106
16:50:55.777058 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.notice, length: 112
16:50:57.704914 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.info, length: 101
16:50:57.817315 IP ***.***.***.***.syslog > logserver.9513: SYSLOG local7.notice, length: 112
16:55:00.701165 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
17:00:00.766060 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 91
17:00:00.777780 IP ***.***.***.***.syslog > logserver.9513: SYSLOG cron.info, length: 79

I know there are few different ways I can use filebeat to read this input data. One is using the default juniper module, which I have enabled. I could also setup a syslog type input in the filebeat.yml config file. I have tried both of these methods using both port 9513 and port 514.

filebeat.inputs:

- type: syslog
  protocol.udp:
    host: "localhost:514"
- type: syslog
  protocol.udp:
    host: "localhost:9513"



- module: juniper
  junos:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
      var.input: udp
      var.syslog_host: localhost
      var.syslog_port: 514
- module: juniper
  junos:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
      var.input: udp
      var.syslog_host: localhost
      var.syslog_port: 9513

I am guessing that there is something in the filebeat configuration that I am not understanding, since filebeat isn't sending any of the syslog info to logstash. I also used tcpdump on port 5044 and the lo interface, the port that the tutorial uses to setup the output of filebeat and the input of logstash. When I run it and receive a new log from the switch on either port 514 or 9513 nothing gets sent through 5044 as shown here:

user@logserver:~$ sudo tcpdump -i lo port 5044
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

Any help understanding filebeat would be much appreciated. Thank you in advanced

Answer 1

I am not sure what I did exactly, but it appears to be working now. I deleted the juniper configuration of "set system syslog host . . . " and redid it without the port assignment. This meant all my syslog messages were now going to the 514 port, so I checked that port on the server. It was receiving all the logs, so I then reset the logstash and filebeat config files to the default settings and disabled all modules on filebeat. I disabled and stopped both filebeat and logstash, re-enabled, and restarted. After that I re-enabled the system and juniper modules in filebeat. This somehow worked and I can now see the syslogs coming in to elasticsearch from the juniper switch.