logstash file input configuration

Question

My simple config looks like this. I have created a dummy folder in my home directory and created some log files in it. My config file looks like this.

input{
    file{
        type => "dummylog"
        path => [/home/rohit/dummy/*.log" ]
    }
}
output{
    elasticsearch{
        embedded => true
    }
}

Now after running logstash i am unable to see in any files in web ui of logstash. Those files have not fetched into elasticsearch. I am using an embedded elasticsearch so no need to run a separate process. Can anyone help me where i am committing mistake?

Answer 1

input {
  file {
    path => "/home/rohit/dummy/*.log"
    type => "log"
  }
}

filter {
  if [type] == "log" {
    grok {
      pattern => "%{COMBINEDAPACHELOG}"
    }
  }
}

output {
   elasticsearch { host => localhost } stdout { } }
}

to check for the pattern if correct run

$ bin/logstash agent -f httpd-shipper.conf --configtest

Answer 2

Perhaps you should fix the syntax errors in your config and then see if there is a deeper problem.

You are missing a " in the line:

    path => [/home/rohit/dummy/*.log" ] 

If that doesn't solve the problem, it would be wise to confirm that the user that runs the logstash process, has read access to the log(s) it is trying to read.

Answer 3

Be very careful of the syntax. You missed "(quotes) in the path

input{
    file{
        type => "dummylog"
        path => ["/home/rohit/dummy/*.log"]
    }
}

output{
    elasticsearch{
        embedded => true
    }
}

Also note that for this to work, elasticsearch has to be on the same host as logstash.

Answer 4

In addition to the opening quotation mark you must have forgotten about:

start_position => ... # string, one of ["beginning", "end"] (optional), default: "end"

So, it should be:

input{
  file{
    type => "dummylog"
    path => "/home/rohit/dummy/*.log"
    start_position => "beginning"
  }
}
  output {
     elasticsearch{
      embedded => true
  }
}