CSV geodata into elasticsearch as a geo_point type using logstash

Question

Below is a reproducible example of the problem I am having using to most recent versions of logstash and elasticsearch.

I am using logstash to input geospatial data from a csv into elasticsearch as geo_points.

The CSV looks like the following:

$ head simple_base_map.csv 
"lon","lat"
-1.7841,50.7408
-1.7841,50.7408
-1.78411,50.7408
-1.78412,50.7408
-1.78413,50.7408
-1.78414,50.7408
-1.78415,50.7408
-1.78416,50.7408
-1.78416,50.7408

I have create a mapping template that looks like the following:

$ cat simple_base_map_template.json 
{
  "template": "base_map_template",
  "order":    1,
  "settings": {
    "number_of_shards": 1
  },

      "mappings": {
        "node_points" : {
          "properties" : {
            "location" : { "type" : "geo_point" }
          }
        }
      }
}

and have a logstash config file that looks like the following:

$ cat simple_base_map.conf 
input {
  stdin {}
}

filter {
  csv {
      columns => [
        "lon", "lat"
      ]
  }

  if [lon] == "lon" {
      drop { }
  } else {
      mutate {
          remove_field => [ "message", "host", "@timestamp", "@version"     ]
      }
       mutate {
          convert => { "lon" => "float" }
          convert => { "lat" => "float" }
          }

      mutate {
          rename => {
              "lon" => "[location][lon]"
              "lat" => "[location][lat]"
          }
      }
  }
}

output {
  stdout { codec => dots }
  elasticsearch {
      index => "base_map_simple"
      template => "simple_base_map_template.json"
      document_type => "node_points"
  }
}

I then run the following:

$cat simple_base_map.csv | logstash-2.1.3/bin/logstash -f simple_base_map.conf 
Settings: Default filter workers: 16
Logstash startup completed
....................................................................................................Logstash shutdown completed

However when looking at the index base_map_simple, it suggests the documents would not have a location: geo_point type in it...and rather it would be two doubles of lat and lon.

$ curl -XGET 'localhost:9200/base_map_simple?pretty'
{
  "base_map_simple" : {
    "aliases" : { },
    "mappings" : {
      "node_points" : {
        "properties" : {
          "location" : {
            "properties" : {
              "lat" : {
                "type" : "double"
              },
              "lon" : {
                "type" : "double"
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1457355015883",
        "uuid" : "luWGyfB3ToKTObSrbBbcbw",
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "2020099"
        }
      }
    },
    "warmers" : { }
  }
}

How would i need to change any of the above files to ensure that it goes into elastic search as a geo_point type?

Finally, I would like to be able to carry out a nearest neighbour search on the geo_points by using a command such as the following:

curl -XGET 'localhost:9200/base_map_simple/_search?pretty' -d'
{
    "size": 1,
    "sort": {
   "_geo_distance" : {
       "location" : {
            "lat" : 50,
            "lon" : -1
        },
        "order" : "asc",
        "unit": "m"
   } 
    }
}'

Thanks

Answer 1

The problem is that in your elasticsearch output you named the index base_map_simple while in your template the template property is base_map_template , hence the template is not being applied when creating the new index. The template property needs to somehow match the name of the index being created in order for the template to kick in.

It will work if you simply change the latter to base_map_* , ie as in:

{
  "template": "base_map_*",             <--- change this
  "order": 1,
  "settings": {
    "index.number_of_shards": 1
  },
  "mappings": {
    "node_points": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}

UPDATE

Make sure to delete the current index as well as the template first., ie

curl -XDELETE localhost:9200/base_map_simple
curl -XDELETE localhost:9200/_template/logstash