Elasticsearch - 搜索通配符（包含在字符串中）和 tf-idf 分数

Question

how can I make a search wildcard and tf-idf scores. example when I search like this,

GET /test_es/_search?explain=true // return idf / dt scores
{
  "explain":true,
  "query": {
    "query_string": {
      "query": "bar^5",
      "fields"  : ["field"]
    }
  }
}

it returns idf and td score, but when I search like with wildcards (contains).

GET /test_es/_search?explain=true  // NOT RETURN idf/td score
{
   "explain":true,
  "query": {
    "query_string": {
      "query": "b*",
      "fields"  : ["field"]
    }
  }
}

how can I make a search with wildcards (using contains in the string) and include the IDF-TD scores?

for example, I have 3 documents " foo ", " foo bar ", " foo baz " when I search it like that

GET /foo2/_search?explain=true
{
   "explain":true,
  "query": {
    "query_string": {
      "query": "fo *",
      "fields"  : ["field"]
    }
  }
}

Elasticsearch Result

    "hits" : [
  {
    "_shard" : "[foo2][0]",
    "_node" : "z8bjI0T1T8Oq6Z2OwFyIKw",
    "_index" : "foo2",
    "_type" : "_doc",
    "_id" : "3",
    "_score" : 1.0,
    "_source" : {
      "field" : "foo bar"
    },
    "_explanation" : {
      "value" : 1.0,
      "description" : "sum of:",
      "details" : [
        {
          "value" : 1.0,
          "description" : "*:*",
          "details" : [ ]
        }
      ]
    }
  },
  {
    "_shard" : "[foo2][0]",
    "_node" : "z8bjI0T1T8Oq6Z2OwFyIKw",
    "_index" : "foo2",
    "_type" : "_doc",
    "_id" : "2",
    "_score" : 1.0,
    "_source" : {
      "field" : "foo"
    },
    "_explanation" : {
      "value" : 1.0,
      "description" : "sum of:",
      "details" : [
        {
          "value" : 1.0,
          "description" : "*:*",
          "details" : [ ]
        }
      ]
    }
  },
  {
    "_shard" : "[foo2][0]",
    "_node" : "z8bjI0T1T8Oq6Z2OwFyIKw",
    "_index" : "foo2",
    "_type" : "_doc",
    "_id" : "1",
    "_score" : 1.0,
    "_source" : {
      "field" : "foo baz"
    },
    "_explanation" : {
      "value" : 1.0,
      "description" : "sum of:",
      "details" : [
        {
          "value" : 1.0,
          "description" : "*:*",
          "details" : [ ]
        }
      ]
    }
  }
]

But I expect "foo" should be the first result with having the highest score because it matches %100, am I wrong?

Answer 1

Since you have not mentioned anything about data that you have taken, I have indexed the following data:

Index data:

{
    "message": "A fox is a wild animal."
}
{
    "message": "That fox must have killed the hen."
}
{
    "message": "the quick brown fox jumps over the lazy dog"
}

Search Query:

GET/{{index-name}}/_search?explain=true 

{
  "query": {
    "query_string": {
      "fields": [
        "message"                       ---> You can add more fields here
      ],
      "query": "quick^2 fox*"
    }
  }
}

The query above searches all the documents containing fox , but here since boost is applied to quick , so the document containing quick fox will have a higher score as compared to other documents.

This query will return the tf-IDF score. The boost operator is used, to make one term more relevant than another.

To know more about this refer to this official documentation on "Boosting" in dsl-query-string

To know more about the tf-IDF algorithm you can refer to this blog

If you want to search across multiple fields, you can boost the scores in a certain field

Refer this and this to know more.

Update 1:

Index Data:

{
  "title": "foo bar"
}
{
  "title": "foo baz"
}
{
  "title": "foo"
}

Search Query:

{
  "query": {
    "query_string": {
      "query": "foo *"         --> You can just add a space between 
                                   foo and *
     }
  }
}

Search Result:

"hits": [
         {
            "_index": "foo2",
            "_type": "_doc",
            "_id": "1",
            "_score": 1.9808292,       --> foo matches exactly, so the 
                                           score is maximum
            "_source": {
               "title": "foo"
            }
         },
         {
            "_index": "foo2",
            "_type": "_doc",
            "_id": "2",
            "_score": 1.1234324,
            "_source": {
               "title": "foo bar"
            }
         },
         {
            "_index": "foo2",
            "_type": "_doc",
            "_id": "3",
            "_score": 1.1234324,
            "_source": {
               "title": "foo baz"
            }
         }
      ]

Answer 2

Update 2:

Wildcard Queries basically falls under Term-level queries, and by default uses the constant_score_boolean method for matching terms.

By changing the value of the rewrite parameter you can impact search performance and relevance. It has various options for scoring, you can choose any of them according to your requirement.

But according to your use case, you may also use edge_ngram filter. Edge N-Grams are useful for search-as-you-type queries. To know more about this and the mapping used below refer to this official documentation

Index Mapping:

{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete": {
          "tokenizer": "autocomplete",
          "filter": [
            "lowercase"
          ]
        },
        "autocomplete_search": {
          "tokenizer": "lowercase"
        }
      },
      "tokenizer": {
        "autocomplete": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 10,
          "token_chars": [
            "letter"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "autocomplete",
        "search_analyzer": "autocomplete_search"
      }
    }
  }
}

Index sample data:

{ "title":"foo" }
{ "title":"foo bar" }
{ "title":"foo baz" }

Search Query:

{
  "query": {
    "match": {
      "title": {
        "query": "fo"
      }
    }
  }
}

Search Result:

"hits": [
            {
                "_index": "foo6",
                "_type": "_doc",
                "_id": "1",
                "_score": 0.15965709,        --> Maximum score
                "_source": {
                    "title": "foo"
                }
            },
            {
                "_index": "foo6",
                "_type": "_doc",
                "_id": "2",
                "_score": 0.12343237,
                "_source": {
                    "title": "foo bar"
                }
            },
            {
                "_index": "foo6",
                "_type": "_doc",
                "_id": "3",
                "_score": 0.12343237,
                "_source": {
                    "title": "foo baz"
                }
            }
        ]

To know more about basics of using Ngrams in Elasticsearch you can referthis