如何从不同服务器收集日志到中央服务器（Elasticsearch 和 kibana）

Question

I am assigned with task to create a central logging server. In my case there are many web app servers spread across. My task is to get logs from these different servers and manage in central server where there will be elastic-search and kibana .

Question

1. Is it possible to get logs from servers that are having different public IP? If possible how?
2. How much resource (CPU, Memory, Storage) is required in central server.

Things seen

• Saw the examples setups where all logs and applications are on same machine only.

Looking for way to send logs over public IP to elastic-search.

Answer 1

I would like to differ from the Ishara's Answer. You can ship logs directly from filebeat to elasticsearch without using logstash, If your logs are generic types(system logs, nginx logs, apache logs), Using this approach You don't need to go into incur extra cost and maintenance of logstash as filebeat provides inbuilt parsing processor.

If you have debian based OS on your server, I have prepared a shell script to install and configure filebeat. You need to change elasticsearch server URL and modify second last line based on the modules that you want to configure.

Regarding your first question, Yes, You can run filebeat agent on each server and send data to centralize Elasticsearch. For your second question, It depends on the amount of logs elasticsearch server is going to process and store. It also depends on the where kibana is hosted.

sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install -y filebeat

sudo systemctl enable filebeat

sudo bash -c  "cat >/etc/filebeat/filebeat.yml" <<FBEOL
filebeat.inputs:

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.name: "filebeat-system"
setup.template.pattern: "filebeat-system-*"
setup.template.settings:
  index.number_of_shards: 1

setup.ilm.enabled: false

setup.kibana:

output.elasticsearch:
  hosts: ["10.32.66.55:9200", "10.32.67.152:9200", "10.32.66.243:9200"]
  indices:
    - index: "filebeat-system-%{+yyyy.MM.dd}"
      when.equals:
        event.module: system

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

logging.level: warning

FBEOL

sudo filebeat modules enable system
sudo systemctl restart filebeat

Answer 2

1. Yes, it is possible to get logs from servers that are having different public IP. You need to setup an agent like filebeat (provided by elastic) to each server which produce logs.

   • You need to setup filebeat instance in each machine.

It will listen to your log files in each machine and forward them to the logstash instance you would mention in filebeat.yml configuration file like below:

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /path_to_your_log_1/ELK/your_log1.log
    - /path_to_your_log_2/ELK/your_log2.log

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["private_ip_of_logstash_server:5044"]

• Logstash server listens to port 5044 and stream all logs through logstash configuration files:

   input { beats { port => 5044 } } filter { # your log filtering logic is here } output { elasticsearch { hosts => [ "elasticcsearch_server_private_ip:9200" ] index => "your_idex_name" } }

• In logstash you can filter and split your logs into fields and send them to elasticsearch.

  2. Resources depend on how much of data you produce, data retention plan, TPS and your custom requirements. If you can provide some more details, I would be able to provide a rough idea about resource requirement.