Elasticsearch - Include fields in highlight excluded in _source

Question

I know objects marked as excluded in the _source mapping can be included in the search query. But I have a requirement to include matching terms in the highlight section of the response.

eg I have a mapping like:

{
  "mappings": {
    "doc": {
      "_source": {
        "excludes": ["some_nested_object.complex_tags_object"]
      },
      "properties": {
        "some_nested_object": {
          "type": "nested"
        }
      }
    }
  }
}

Search Query:

GET my_index/_search {
    "size": 500,
    "query": {
        "bool": {
            "must": [{
                    "nested": {
                        "query": {
                            "bool": {
                                "must": 
                                [{
                                    "match_phrase_prefix": {
                                        "some_nested_object.complex_tags_object.name": {
                                            "query": "account"
                                        }
                                    }
                                }
                                ]
                            }
                        },
                        "path": "some_nested_object"
                    }
                }
            ]
        }
    },  
    "highlight": {
        "pre_tags": [
            ""
        ],
        "post_tags": [
            ""
        ],
        "fields": {
            "some_nested_object.complex_tags_object.name": {}
        }
    }
}

If I don't exclude in the mapping but in the search query at runtime then I am able to return matching terms in the highlight section but the response is very slow due to the large size of the object.

So is it possible to include fields marked as exclude in the mapping/doc/_source as part of highlight ?

Answer 1

So is it possible to include fields marked as exclude in the mapping/doc/_source as part of highlight?

The short answer to your question unfortunately is no . From the Elasticsearch highlighting documentation :

Highlighting requires the actual content of a field. If the field is not stored (the mapping does not set store to true ), the actual _source is loaded and the relevant field is extracted from _source .

You have a few options, each of which involve compromise:

• Include your field back into the source if you absolutely need to support highlighting over it (I appreciate this will conflict with the reasons for excluding it from the source in the first place)
• Relax the requirement to support highlighting over this field (compromise on features)
• Implement a highlighting feature for this field outside Elasticsearch (probably this will compromise on quality of your solution and perhaps cost)