如何使用 node.js 设置 ELK

Question

I want to log error from my node.js server to another server. I'm thinking of using elasticsearch, logstash and kibana. I want to know how to setup ELK with my node server.

Answer 1

I had exactly this use case in my older organization. A basic tutorial to startup with Beats + ELK - https://www.elastic.co/guide/en/beats/libbeat/current/getting-started.html

So basically this is how it works - Your nodejs app will log in the files (you can use bunyan for this) in different formats like error/warning/info etc. Filebeat will tail these log files and send messages to logstash server. Logstash input.conf will have some input filters (in your case it will be error filters). When any log message passes these filters then logstash will forward it to some endpoint as decided in output.conf file.

Here is what we did -

Initial architecture - Install filebeat (earlier we used logstash forwarder) client to tail the logs on nodejs server and forward it to logstash machine. Logstash will do some processing on input logs and send them to ES cluster (can be on same machine as Logstash). Kibana is just a visualization on this ES.

Final Architecture - Initial setup was cool for small traffic but we realized that logstash can be single point of failure and may result in log loss when traffic increased. So we integrated Kafka along with Logstash so that system scales smoothly. Here is an article - https://www.elastic.co/blog/logstash-kafka-intro

Hope this helps!

Answer 2

It is possible to use logstash without agents running to collect logs from the application. Logstash has input plugins ( https://www.elastic.co/guide/en/logstash/current/input-plugins.html ). This can be configured in the pipeline. One basic setup is to configure the TCP ( https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html ) or UDP ( https://www.elastic.co/guide/en/logstash/current/plugins-inputs-udp.html)input plugin. Logstash can listen on the port configured in the plugin. Then the application can send the log directly to the logstash server. The pipeline can then transform and forward to ES. By configuring Logstash pipeline to be durable, data loss can be avoided. This approach is better when the application servers are ephemeral ( as in containers).

For nodejs, https://www.npmjs.com/package/winston-logstash is a package which is quite active. This gist https://gist.github.com/jgoodall/6323951 provides a good example for the overall approach in other languages.

This is the sample (minimal) TCP input plugin configuration

input {
  tcp {
  'port' => '9563'
  }
}

Answer 3

You can install Logstash in the NodeJS Server, and then create a configuration file that accepts input (location of the log file(s)) and output to your Elastic Server host.

Below is the sample configuration file (custom.conf) which has to created in your logstash directory.

    input {
    file {
    path => "/path to log"
    start_position => beginning
    }
    }
    output {
    stdout { }
    elasticsearch{
    type => "stdin-type"
    embedded => false
    host => "192.168.0.23"
    port => "9300"
    cluster => "logstash-cluster"
    node_name => "logstash"
    }
    }

Execute the logstash

    logstash -f custom.conf

Reference: https://www.elastic.co/guide/en/logstash/current/config-examples.html

If you are planning to customize a NodeJS application for sending error logs then you can install some ELKStack Nodjs wrapper and post error log within your application. ELKStack Nodjs wrapper - https://www.npmjs.com/package/elksdk