如何在两台分离的服务器之间使用Logstash将日志从文件发送到Elasticsearch？

Question

I try to send logs from logs.csv file to elasticsearch using Logstash. In Elasticsearch I have index logs with type log. At the moment my logstash.conf looks in this way:

input {
  file {
    path => "/run/shm/elastic/logstash/logs.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
      columns => ["logs"]
  }

}
output {
    elasticsearch {
        hosts => "hostaddress:9200"
        index => "logs"
        document_type => "log"
        user => "elastic"
        password => "elastic"
    }
    stdout {}
}

Logstash seems to be configured correctly because for instance sudo ./logstash -e 'input { stdin { } } output { stdout {} }' works properly. However I get error shown below. Any ideas?

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2018-07-11 10:48:27.473 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2018-07-11 10:48:27.510 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[ERROR] 2018-07-11 10:48:27.522 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Answer 1

This error happens because another instance of Logstash is still running. You should start Logstash as a service in Linux instead of directly starting it, for example on RHEL you should start using:

service logstash start

and stop

service logstash stop 

You can find commands for other systems under this link .

But sometimes Logstash gets stalled and you have to kill it manually

ps aux | grep logstash

Find Logstash's PID and kill it:

kill -9 LOGSTASH_PID

Most of the time Logstash can't be stopped in the standard way because it's processing some data but you can force Logstash to stop by adding --pipeline.unsafe_shutdown in the service startup file, you can read more about this here .

Answer 2

If you want to run multiple logstash instances, you need to define the path.data either by command,

bin/logstash -f <config_file.conf> --path.data PATH

(make sure the directory is writable)

or specify in logstash.yml file under /etc/logstash/ for each instance.

Please read, logstash could not be started when running multiple instances - path.data setting

To read all columns fro csv file, you need to provide name of each column like this,

columns => ["Date","column2","column3"]