简体   繁体   中英

Interpret locations from Keen.io JSON file in logstash filter

I'm trying to parse a JSON file from Keen.io with logstash into elasticsearch. The location and timestamp are stored in parameters like this:

{
  "result":
  [
    {
      "keen":
      {
        "timestamp": "2014-12-02T12:23:51.000Z",
        "created_at": "2014-12-01T23:25:31.396Z",
        "id": "XXXX",
        "location":
        {
          "coordinates": [-95.8, 36.1]
        }
      }
    }
  ]
}

My filter currently looks like this:

input {
  file {
    path => ["test.json"]
    start_position => beginning
    type => json
  }
}

filter {
  json {
    source => message
    remove_field => message
  }
}

output {
  stdout { codec => rubydebug }
}

How can I parse the "timestamp" and "location" fields so they are used for the @timestamp and @geoip.coordinates in Elasticsearch?

Update: I've tried variations of this with no luck. The documentation is very basic - am I misunderstanding how to reference the JSON fields? Is there a way of adding debug output to help? I tried How to debug the logstash file plugin and Print a string to stdout using Logstash 1.4? but neither works.

filter {
  json {
    source => message
    remove_field => message
  }
  if ("[result][0][keen][created_at]") {
    date {
      add_field => [ "[timestamp]", "[result][0][keen][created_at]" ]
      remove_field => "[result][0][keen][created_at]"
    }
  }

Update 2:

Date is working now, still need to get location working.

filter {
  json {
    source => message
    remove_field => message
    add_tag => ["valid_json"]
  }
  if ("valid_json") {
    if ("[result][0][keen][created_at]") {
      date {
        match => [ "[result][0][keen][created_at]", "ISO8601" ]
      }
    }
  }
}

Keen.io's "created_at" field is stored in ISO 8601 format and so can easily be parsed by the date filter. Lat/long co-ordinates can be set by copying Keen.io's existing co-ordinates into logstash's geoip.coordinates array.

input {
  file {
    path => ["data.json"]
    start_position => beginning
    type => json
  }
}

filter {
  json {
    source => message
    remove_field => message
    add_tag => ["valid_json"]
  }
  if ("valid_json") {
    if ("[result][0][keen][created_at]") {
      date {
        # Set @timestamp to Keen.io's "created_at" field
        match => [ "[result][0][keen][created_at]", "ISO8601" ]
      }
    }
    if ("[result][0][keen][location][coordinates]") {
      mutate {
        # Copy existing co-orndiates into geoip.coordinates array
        add_field => [ "[geoip][coordinates]", "%{[result][0][keen][location][coordinates][0]}" ]
        add_field => [ "[geoip][coordinates]", "%{[result][0][keen][location][coordinates][1]}" ]
        remove_field => "[result][0][keen][location][coordinates]"
      }
    }
  }
}

output {
  stdout { codec => rubydebug }
}

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM