Using Elasticsearch filter in logstash

Question

I'm trying to use the elastic search filter on logstash for make some Data Enrichment.

I got two indexes, and my goal it's get some data from one of them and add it to the other.

I configured a logstash filter who search in my elasticsearch and if there is a match the output goes to the index.

But my filter it's not working propery because when I test the filter i got this error

[WARN ] 2020-10-02 19:23:09.536 [[main]>worker2] elasticsearch - Failed to query elasticsearch for previous event {:index=>"logstash-*", :error=>"Unexpected character ('%' (code 37)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n

I think there it's some issue between the variable in the template and the elastic search

my logstash it's a 7.3.2 and my ES an 7.4.2

here it's my settings

       Logstash.conf
        
       input {
               http{ }
       }
        
       filter {
               elasticsearch {
                 hosts => ["127.0.0.1:9200"]
                 index => "logstash-*"
                 query_template => "search-by-ip.json"
                 fields => {
                         "id"  => "[suscriberid]"
                 }
               }
         }
        
        
       output {
         stdout { codec => rubydebug }
       }
        
       -----------------
       search-by-ip.json
        
       {
         "size": 1,
         "query": { "match":{"IP": %{[ip]} } }
       }
       -------------------
       testcase.sh
        
       curl -XPOST "localhost:8080" -H "Content-Type: application/json" -d '{
         "size": 1,
         "query": { "match":{"ip": "192.168.1.4" }}
       }'
        
      ```  



Thanks!

Answer 1

If you ever process an event that does not have an [ip] field then the sprintf reference will not be substituted and you will get that error.

Note that ip and IP are different fields. Not sure if the %{[ip]} requires double quotes around it.