如何在 Logstash Conf 文件的地理坐標中刪除引號（“”）

Question

我想在 Kibana 的 map 上表達一條線，而 map 應該是 Geojson 結構。 我擁有的數據是一組 SQL 表，然后我打算像這樣使用 Logstash 將它們傳輸到 Elastic Search

input{ ... }

filter{
    
 if [lat] and [lon] {
        mutate{convert =>  ["lat", "float"]}
        mutate{convert =>  ["lon", "float"]}
        mutate{convert =>  ["for_lat", "float"]}
        mutate{convert =>  ["for_lon", "float"]}

        mutate{
            add_field => {"[location-geotest][type]" => "multilinestring"}
            add_field => {"[location-geotest][coordinates]" => [["%{lon}", "%{lat}"]]}
            add_field => {"[location-geotest][coordinates]" => [["%{for_lon}", "%{for_lat}"]]}
            }
      }
}

但是，logstash conf 文件未能索引 Elasticsearch 上的數據

{
    :status=>400, 
    :action=>["index", {:_id=>"18022", :_index=>"geo_shape_test", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x687994b9>], 
    :response=> {
        "index"=>{
            "_index"=>"geo_shape_test", 
            "_type"=>"_doc", 
            "_id"=>"18022", 
            "status"=>400, 
            "error"=>{
                "type"=>"mapper_parsing_exception", 
                "reason"=>"failed to parse field [location-geotest] of type [geo_shape]", 
                "caused_by"=>{"type"=>"x_content_parse_exception", 
                    "reason"=>"[1:164] [geojson] failed to parse field [coordinates]", 
                    "caused_by"=>{
                        "type"=>"parse_exception", 
                        "reason"=>"geo coordinates must be numbers"
                    }
                }
            }
        }
    }
}

這是logstash試圖索引的內容之一

{
                 "lat" => 37.567179953757886,
              "gps_id" => 10491,
           "timestamp" => 2020-11-22T06:10:45.000Z,
               "speed" => 17.25745240090587,
                 "lon" => 126.99598717854032,
             "for_lat" => 37.567179953757886,
          "@timestamp" => 2020-11-27T03:54:21.131Z,
             "for_lon" => 126.99598717854032,
            "@version" => "1",
    "location-geotest" => {
        "coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],
               "type" => "multilinestring"
    }
}

我認為問題是...

 "coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],

如果我改變部分，那將是......

 "coordinates" => [
            [0] [126.99598717854032, 37.567179953757886],
            [1] [126.99598717854032, 37.567179953757886]
        ],

但我找不到如何解決。

Answer 1

我認為問題是正如你所說，坐標必須是浮點數而不是字符串。 顯然，變異 function 將值轉換回字符串。 如中所述

https://discuss.elastic.co/t/logstash-mutate-filter-always-stringifies-hash-and-array/25917

他們建議使用 ruby 腳本的解決方案。 這已經為線串完成了。

https://discuss.elastic.co/t/geo-shape-geo-link-problems-with-coordinates/179924/4

從提供的數據中，我看不出為什么需要多行字符串？ 只有兩個點就足以存儲為線串。

我試過了

filter{
    
 if [lat] and [lon] {

        mutate{
            convert =>  ["lat", "float"]
            convert =>  ["lon", "float"]
            convert =>  ["for_lat", "float"]
            convert =>  ["for_lon", "float"]
            
            add_field => {"[location-geotest][type]" => "linestring"}
        }

      ruby{
          code => "event.set('[location-geotest][coordinates]', [[event.get('lon'), event.get('lat')], [event.get('for_lon'), event.get('for_lat')]])"
      }
  }
}

並得到結果：

    "location-geotest" => {
           "type" => "linestring",
           "coordinates" => [
                 [0] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ],
                 [1] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ]
           ]
         }

哪個索引正確。

如果您需要多字符串，我想您需要更多數據並在 ruby 腳本中添加一層 arrays。