在 Logstash 中过滤 jdbc 数据

Question

In my DB, I've data in below format:

But in ElasticSearch I want to push data with respect to item types. So each record in ElasticSearch will list all item names & its values per item type.

Like this:

{
  "_index": "daily_needs",
  "_type": "id",
  "_id": "10",
  "_source": {
    "item_type: "10",
    "fruits": "20",
    "veggies": "32",
    "butter": "11",
  }
}

{
  "_index": "daily_needs",
  "_type": "id",
  "_id": "11",
  "_source": {
    "item_type: "11",
    "hair gel": "50",
    "shampoo": "35",
  }
}

{
  "_index": "daily_needs",
  "_type": "id",
  "_id": "12",
  "_source": {
    "item_type: "12",
    "tape": "9",
    "10mm screw": "7",
    "blinker fluid": "78",
  }
}

Can I achieve this in Logstash ?

I'm new into Logstash, but as per my understanding it can be done in filter . But I'm not sure which filter to use or do I've to create a custom filter for this.

Current conf example:

input {
  jdbc {
    jdbc_driver_library => "ojdbc6.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    jdbc_connection_string => "myjdbc-configs"
    jdbc_user => "dbuser"
    jdbc_password => "dbpasswd"
    schedule => "* * * * *"
    statement => "SELECT * from item_table"
  }
}
filter {
    ## WHAT TO WRITE HERE??
}
output {
    elasticsearch {
        hosts => [ "http://myeshost/" ]
        index => "myindex"
    }
}

Kindly suggest. Thank you.

Answer 1

You can achieve this using aggregate filter plugin . I have not tested below, but should give you an idea.

 filter {     
      aggregate {
        task_id => "%{item_type}" #
        code => "
          map['Item_type'] = event.get('Item_type')
          map[event.get('Item_Name')] = map[event.get('Item_Value')]
        "
        push_previous_map_as_event => true
        timeout => 3600
        timeout_tags => ['_aggregatetimeout']
      }
      if "aggregated" not in [tags] {
        drop {}
      }
    }

Important Caveats for using aggregate filter :

• The sql query MUST order the results by Item_Type, so the events are not out of order.
• Column names in sql query should match the column names in the filter map[]
• You should use ONLY ONE worker thread for aggregations otherwise events may be processed out of sequence and unexpected results will occur.