带有Nest字段的查询提升没有从Elasticsearch返回结果

Question

I'm having real problems getting a query using field boosting to work with Elasticsearch. I've been through the Nest docs on the topic, but they weren't particularly helpful, so my code is really based on the solution to this question: Elastic Search using NEST Field Boosting .

If I run the following query I get one result, as expected:

var matches =
    _client.Search<SearchableMerchant>(
        s => s.From((page - 1) * pageSize)
            .Size(pageSize)
            .QueryString("*test*")
            .MinScore(1)
    );

However, if I try to use field boosting, using the following, I get no matches:

var matches =
            _client.Search<SearchableMerchant>(
                s => s.From((page - 1) * pageSize)
                    .Size(pageSize)
                    .Query(q => q
                        .Boosting(bq => bq
                            .Positive(pq => pq
                                .CustomScore(cbf => cbf
                                    .Query(cbfq => cbfq
                                        .QueryString(
                                            qs => qs
                                            .OnFieldsWithBoost(d => d
                                                .Add("opportunities.acquirerLocationMID", Math.Pow(2, 17))
                                                .Add("opportunities.amexMID", Math.Pow(2, 16))
                                                .Add("opportunities.epayMID", Math.Pow(2, 16))
                                                .Add("v1MerchantId", Math.Pow(2, 16))
                                                .Add("locatorId", Math.Pow(2, 15))
                                                .Add("opportunities.opportunityLocatorId", Math.Pow(2, 14))
                                                .Add("businessName", Math.Pow(2, 13))
                                                .Add("searchablePhone", Math.Pow(2, 12))
                                                .Add("address.postCodeDetails.postCode.postCode", Math.Pow(2, 11))
                                                .Add("contacts.contact.searchableEmailAddress", Math.Pow(2, 11))
                                                .Add("contacts.contact.searchableMainPhone", Math.Pow(2, 10))
                                                .Add("contacts.contact.searchableMobilePhone", Math.Pow(2, 10))
                                                .Add("contacts.contact.fullName", Math.Pow(2, 9))
                                                .Add("contacts.contact.surname", Math.Pow(2, 8))
                                                .Add("contacts.contact.firstName", Math.Pow(2, 7))
                                                .Add("searchableAddress", Math.Pow(2, 6))
                                                .Add("ownershipUser.username", Math.Pow(2, 5))
                                                .Add("ownershipUser.searchableFullName", Math.Pow(2, 4))
                                                .Add("ownershipUser.lastName", Math.Pow(2, 3))
                                                .Add("ownershipUser.firstName", Math.Pow(2, 2))
                                                .Add("opportunities.depositAccount", Math.Pow(2, 1))
                                                .Add("opportunities.depositIban", Math.Pow(2, 1))
                                                .Add("opportunities.feesAccount", Math.Pow(2, 1))
                                                .Add("opportunities.feesIban", Math.Pow(2, 1))
                                                //  TODO: Company registration number - somewhere in legal methinks
                                            )
                                            .Query(
                                                "*test*"
                                            )
                                        )
                                    )
                                )
                            )
                            .Negative(nq => nq
                                .Filtered(nfq => nfq
                                    .Query(qq => qq.MatchAll())
                                    .Filter(f =>
                                        f.Missing("opportunities.acquirerLocationMID")
                                        && f.Missing("opportunities.amexMID")
                                        && f.Missing("opportunities.epayMID")
                                        && f.Missing("v1MerchantId")
                                        && f.Missing("locatorId")
                                        && f.Missing("opportunities.opportunityLocatorId")
                                        && f.Missing("businessName")
                                        && f.Missing("searchablePhone")
                                        && f.Missing("address.postCodeDetails.postCode.postCode")
                                        && f.Missing("contacts.contact.searchableEmailAddress")
                                        && f.Missing("contacts.contact.searchableMainPhone")
                                        && f.Missing("contacts.contact.searchableMobilePhone")
                                        && f.Missing("contacts.contact.fullName")
                                        && f.Missing("contacts.contact.surname")
                                        && f.Missing("contacts.contact.firstName")
                                        && f.Missing("searchableAddress")
                                        && f.Missing("ownershipUser.username")
                                        && f.Missing("ownershipUser.searchableFullName")
                                        && f.Missing("ownershipUser.lastName")
                                        && f.Missing("ownershipUser.firstName")
                                        && f.Missing("opportunities.depositAccount")
                                        && f.Missing("opportunities.depositIban")
                                        && f.Missing("opportunities.feesAccount")
                                        && f.Missing("opportunities.feesIban")
                                    )
                                )
                            )
                            .NegativeBoost(0.01)
                        )
                    )
                    .MinScore(1)
            );

I realise this code could be better structured, but right now I just want to get a field boosting query working - I'll tidy it later.

Here are some things I've tried:

1. The Nest docs are silent on the topic of whether you can use OnFieldsWithBoost with property names. Ie, is this OK?

   .OnFieldsWithBoost(d => d .Add("businessName", Math.Pow(2, 13))

As opposed this?

.OnFieldsWithBoost(d => d
    .Add(m => m.businessName, Math.Pow(2, 13))

The reason I ask is that I have sub-properties I want to boost that sit within collections. For example, opportunities.opportunityLocatorId . Opportunities obviously being the collection, and I want to match where any object in that collection has a matching value for its opportunityLocatorId field.

This works with fields - you can use a lambda or a string - but does it work with boosting?

No idea, but I've tried it both ways, slimming down the query to just include a boost for businessName , since this is the field that should match the string 'test', but still no results come back.

I've also tried getting rid of the .Negative clause, just in case that was matching something it shouldn't. It's there to deboost any query where no match is found in any of the fields listed in the .Positive clause. Still no results.

I've also upped the .NegativeBoost value to 1 (ie, no effect, so any results shouldn't be filtered down to a score below 1 that didn't start out with such a low score) but, again, no dice.

Here's the content of my index, just so you can see that the businessName field should match 'test' with the second query, as it does with the first:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "merchantv2",
      "_type" : "searchablemerchant",
      "_id" : "00000000-0000-0000-0000-000000000000",
      "_score" : 1.0,
      "_source":{"merchantGuid":"00000000-0000-0000-0000-000000000000","v1MerchantId":0,"locatorId":"0","address":{"addressGuid":"00000000-0000-0000-0000-000000000000","postCodeDetails":{"postCodeKey":0,"postalDistrict":{"postalDistrictKey":0,"postalDistrict":""},"postalLocation":"0","latitude":0.0,"longitude":0.0,"townName":"None","countyKey":0,"countryKey":0,"postCode":{"postCodeKey":0,"postCode":" 0"}},"county":{"countyKey":0,"countyName":"","countryKey":0,"recStatus":3,"countryKeyValue":0},"countryKey":0,"addressTypeKey":0,"updateDate":"0001-01-01T00:00:00+00:00","createdDate":"2016-01-07T19:46:28.4463+00:00"},"searchableAddress":" 0","searchablePhone":"","searchableFax":"","businessName":"","contacts":[],"opportunities":[{"opportunityGuid":"00000000-0000-0000-0000-000000000000","merchantGuid":"00000000-0000-0000-0000-000000000000","location":{"locationGuid":"00000000-0000-0000-0000-000000000000","tradingAddress":{"verified":false,"addressGuid":"00000000-0000-0000-0000-000000000000","postCodeDetails":{"postCodeKey":0,"postalDistrict":{"postalDistrictKey":0,"postalDistrict":""},"postalLocation":"0","latitude":0.0,"longitude":0.0,"townName":"None","countyKey":0,"countryKey":0,"postCode":{"postCodeKey":0,"postCode":" 0"}},"county":{"countyKey":0,"countyName":"","countryKey":0,"recStatus":3,"countryKeyValue":0},"countryKey":0,"addressTypeKey":0,"updateDate":"0001-01-01T00:00:00+00:00","createdDate":"2016-01-07T19:46:28.4463+00:00"}},"opportunityLocatorId":"000000"}]}
    }, {
      "_index" : "merchantv2",
      "_type" : "searchablemerchant",
      "_id" : "5f55fe61-ca65-e411-93f3-0cc47a07ef4a",
      "_score" : 1.0,
      "_source":{"merchantGuid":"5f55fe61-ca65-e411-93f3-0cc47a07ef4a","locatorId":"PM227Z02","address":{"addressGuid":"5c55fe61-ca65-e411-93f3-0cc47a07ef4a","houseNumber":"242","streetName":"Acklam Road","houseName":"","flatAptSuite":"","townName":"London","postCodeDetails":{"postCodeKey":1,"postalDistrict":{"postalDistrictKey":2782,"postalDistrict":"W10"},"postalLocation":"5JJ","latitude":51.52094651,"longitude":-0.20149990,"townName":"London","countyKey":0,"countryKey":224,"postCode":{"postCodeKey":1,"postCode":"W10 5JJ"}},"county":{"countyKey":626,"countyName":"Kensington And Chelsea","countryKey":224,"recStatus":1,"countryKeyValue":224},"countryKey":224,"addressTypeKey":0,"updateDate":"0001-01-01T00:00:00+00:00","createdDate":"2016-01-07T19:46:28.4653+00:00"},"searchableAddress":"242 Acklam Road, London, Kensington And Chelsea, W10 5JJ","searchablePhone":"+44 2031954484","searchableFax":"","businessName":"Test Merchant","contacts":[],"opportunities":[]}
    } ]
  }
}

I'm using Elasticsearch 1.7.1 and Nest 1.7.1 on Windows 7 (yeah, I know, but it's what the client uses) with .NET 4.5.1.

I've also tried capturing the traffic going between my Web API and elasticsearch, but to no avail. Possibly a config issue but neither Fiddler nor Wireshark/npcap is able to capture the traffic between these two, both running on the local machine, so I can't see the actual request being sent to elasticsearch, which I suspect would be helpful. Basically I was wondering if any error was coming back from Elasticsearch that Nest was swallowing.

Well... the intuition turned out to be correct. Here's a sample of what's appearing in the elasticsearch log file:

[2016-01-08 10:14:01,534][DEBUG][action.search.type       ] [Rocket Racer] All shards failed for phase: [query]
org.elasticsearch.search.SearchParseException: [user][4]: from[0],size[20]: Parse Failure [Failed to parse source [{
  "from": 0,
  "size": 20,
  "min_score": 1.0,
  "query": {
    "boosting": {
      "positive": {
        "custom_score": {
          "query": {
            "query_string": {
              "query": "*test*",
              "fields": [
                "opportunities.acquirerLocationMID^131072",
                "opportunities.amexMID^65536",
                "opportunities.epayMID^65536",
                "v1MerchantId^65536",
                "locatorId^32768",
                "opportunities.opportunityLocatorId^16384",
                "businessName^8192",
                "searchablePhone^4096",
                "address.postCodeDetails.postCode.postCode^2048",
                "contacts.contact.searchableEmailAddress^2048",
                "contacts.contact.searchableMainPhone^1024",
                "contacts.contact.searchableMobilePhone^1024",
                "contacts.contact.fullName^512",
                "contacts.contact.surname^256",
                "contacts.contact.firstName^128",
                "searchableAddress^64",
                "ownershipUser.username^32",
                "ownershipUser.searchableFullName^16",
                "ownershipUser.lastName^8",
                "ownershipUser.firstName^4",
                "opportunities.depositAccount^2",
                "opportunities.depositIban^2",
                "opportunities.feesAccount^2",
                "opportunities.feesIban^2"
              ]
            }
          }
        }
      },
      "negative": {
        "filtered": {
          "query": {
            "match_all": {}
          },
          "filter": {
            "bool": {
              "must": [
                {
                  "missing": {
                    "field": "opportunities.acquirerLocationMID"
                  }
                },
                {
                  "missing": {
                    "field": "opportunities.amexMID"
                  }
                },
                {
                  "missing": {
                    "field": "opportunities.epayMID"
                  }
                },
                {
                  "missing": {
                    "field": "v1MerchantId"
                  }
                },
                {
                  "missing": {
                    "field": "locatorId"
                  }
                },
                {
                  "missing": {
                    "field": "opportunities.opportunityLocatorId"
                  }
                },
                {
                  "missing": {
                    "field": "businessName"
                  }
                },
                {
                  "missing": {
                    "field": "searchablePhone"
                  }
                },
                {
                  "missing": {
                    "field": "address.postCodeDetails.postCode.postCode"
                  }
                },
                {
                  "missing": {
                    "field": "contacts.contact.searchableEmailAddress"
                  }
                },
                {
                  "missing": {
                    "field": "contacts.contact.searchableMainPhone"
                  }
                },
                {
                  "missing": {
                    "field": "contacts.contact.searchableMobilePhone"
                  }
                },
                {
                  "missing": {
                    "field": "contacts.contact.fullName"
                  }
                },
                {
                  "missing": {
                    "field": "contacts.contact.surname"
                  }
                },
                {
                  "missing": {
                    "field": "contacts.contact.firstName"
                  }
                },
                {
                  "missing": {
                    "field": "searchableAddress"
                  }
                },
                {
                  "missing": {
                    "field": "ownershipUser.username"
                  }
                },
                {
                  "missing": {
                    "field": "ownershipUser.searchableFullName"
                  }
                },
                {
                  "missing": {
                    "field": "ownershipUser.lastName"
                  }
                },
                {
                  "missing": {
                    "field": "ownershipUser.firstName"
                  }
                },
                {
                  "missing": {
                    "field": "opportunities.depositAccount"
                  }
                },
                {
                  "missing": {
                    "field": "opportunities.depositIban"
                  }
                },
                {
                  "missing": {
                    "field": "opportunities.feesAccount"
                  }
                },
                {
                  "missing": {
                    "field": "opportunities.feesIban"
                  }
                }
              ]
            }
          }
        }
      },
      "negative_boost": 0.01
    }
  }
}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.index.query.QueryParsingException: [user] No query registered for [custom_score]
    at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:303)
    at org.elasticsearch.index.query.BoostingQueryParser.parse(BoostingQueryParser.java:63)
    at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:305)
    at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
    at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
    at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
    at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
    ... 9 more

So what am I doing wrong? Does anyone know how to fix the second query, which elasticsearch clearly doesn't like? And is there some way of getting any erorrs out of Nest? I'd expect an exception, but that's not happening - it just returns silently with an empty match collection, and there's no property on the collection that indicates something went wrong.

Any help gratefully received.

Thanks!

Bart

Answer 1

The custom score query was deprecated in Elasticsearch 0.90.4 and removed in Elasticsearch 1.x. It's kept in NEST for backwards compatibility. Instead, you should use the function score query .

NEST however should have indicated that an error occurred via the IsValid property, which should be false in this case. By default, NEST 1.x doesn't throw on Elasticsearch exceptions. You can enable this behavior by setting ThrowOnElasticsearchServerExceptions() on your ConnectionSettings .

Side note: Using a wildcard in the beginning of the term (eg *test ) is generally bad practice as it will cause every single term in the index to be examined. You may want to look into modifying your mappings and use something like the nGram tokenizer instead.

Answer 2

Turns out that what I'm trying to do is pretty simple, and I'd just disappeared down the wrong rabbit hole for a while. For example, here's a multi_match query to which I've applied field boosting:

curl -XGET http://localhost:9200/merchantv2/_search -d '
{
    "query": {
        "multi_match": {
            "query": "test",
            "type": "phrase_prefix",
            "fields" : ["businessName^3", "address.streetName"]
        }
    }
}'

In this case I've boosted the businessName field such that matches found in it are three times as important as those found in address.streetName . Seems to be working just fine.

Here's a link to the relevant documentation: https://www.elastic.co/guide/en/elasticsearch/reference/1.7/query-dsl-multi-match-query.html (props to Val for this, which he suggested for a different question).

Thanks for the pointers!