在Logstash conf中建立索引时如何添加文件名

Question

I have two files named access_log and http_access_2015-03-06_log and I want to set access and http_access_2015-03-06 parts of the file as the indices.

I read some answers for similar questions but I couldn't get how I can filter the file path using grok filter and use it as a reference for indexing.

Below is my configuration file:

input {
  file {
    path => ["G:/logstash-1.5.0/bin/tmp/*_log"]
    start_position => "beginning"

  }
}

filter {
  if [path] =~ "access" {
    mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  } else if [path] =~ "error" {
    mutate { replace => { type => "apache_error" } }
  } else {
    mutate { replace => { type => "random_logs" } }
  }
}

output {
  elasticsearch { 
     action => "index"
     host => localhost
     index => "test" 
  }
  stdout { codec => rubydebug }
}

How it can be done?

Answer 1

You're setting a 'type' field. If that's good enough, you can refer to it in your output{} stanza:

index => "%{type}-%{+YYYY.MM.dd}"

If you wanted to use some information from your filters that you didn't also want stored in elasticsearch, you can put it in a metadata field, eg:

[@metadata][myField]

and then refer to that in the same way.