使用Logstash输出csv插件从ElasticSearch输出文档元数据

Question

I am attempting to output the _id metadata field from ES into a CSV file using Logstash.

{
  "_index": "data",
  "_type": "default",
  "_id": "vANfNGYB9XD0VZRJUFfy",
  "_version": 1,
  "_score": null,
  "_source": {
    "vulnid": "CVE-2018-1000060",
    "product": [],
    "year": "2018",
    "month": "02",
    "day": "09",
    "hour": "23",
    "minute": "29",
    "published": "2018-02-09T18:29:02.213-05:00",
  },
  "sort": [
    1538424651203
  ]
}

My logstash output filter is:

output { csv {  fields => [ "_id", "vulnid", "published"]  path =>
"/tmp/export.%{+YYYY-MM-dd-hh-mm}.csv" } }

I get output:

,CVE-2018-1000060,2018-02-09T18:29:02.213-05:00

But I would like to get:

vANfNGYB9XD0VZRJUFfy,CVE-2018-1000060,2018-02-09T18:29:02.213-05:00

How to output the metadata _id into the csv file? It does not matter if I specify the field like "_id" or "@_id" or "@id".

Answer 1

When we query ES we have to enable docinfo => true. By default it is false.

input {
 elasticsearch {
  hosts => [ your hosts ]
  index => "ti"
  query => '{your query}'
  size => 1000
  scroll => "1s"
  docinfo => true
  schedule => "14 * * * *"
 }
}

Answer 2

Well logstash is not able to get "_id" field from your input, because you must not have set the option docinfo into true.

docinfo helps to include elasticsearch documents information such as index,type _id etc..Please have a look here for more info https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-docinfo

use your input plugin as

input {
  elasticsearch {
    hosts => "hostname"
    index => "yourIndex"
    query => '{ "query": { "query_string": { "query": "*" } } }' //optional
    size => 500 //optional
    scroll => "5m" //optional
    docinfo => true
  }
}