ElasticSearch NEST Checking for null value

Question

I have a DateTime? field, I want to return items if that field is in the future or if its NULL.

The NULL check is the problem, ES doesn't store null values, so I can't check it.

Is there not some way of checking for none existence in the .Query?

I know I can .Filter() to get ES to return items that don't have a specific field, but I need to check for NULL in the .Query() which isn't working.

What I have is:

var results = client.Search<ElasticResult>(s => s
  .Filter(f => f.Missing(ff=>ff.EndTimeUTC) || f.Exists(ff=>ff.EndTimeUTC))         
  .Query(q => q
  .Term(p => p.ShortDescription, "somevalue")
   && ( q.Range(p => p.OnField(f => f.EndTimeUTC).GreaterOrEquals(DateTime.UtcNow)) || 
        q.Term(t => t.EndTimeUTC, null) )   // THIS IS HAVING NO EFFECT
));

I'm not sure that the

 .Filter(f => f.Missing("endTimeUTC") || f.Exists("endTimeUTC"))

Is actually making any difference, the required documents are being returned by the ShortDescription query, they just don't have a endTimeUTC field

Answer 1

I guess, your

 .Filter(f => f.Missing("endTimeUTC") || f.Exists("endTimeUTC"))

makes no sense, as it filters missing || exists missing || exists , so it filters nothing.

If you need to search by range and and the same time show documents without such field, that's what you need

POST so/t4/
{"time": "1900-01-01"}
POST so/t4/
{"time": "2100-01-01"}
POST so/t4
{}

GET so/t4/_search
{
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "bool": {
          "should": [
            {
              "range": {
                "time": {
                  "gte": "now"
                }
              }
            },
            {
              "missing": {
                "field": "time"
              }
            }
          ]
        }
      }
    }
  }
}

results in:

{
   "took": 2,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 2,
      "max_score": 1,
      "hits": [
         {
            "_index": "so",
            "_type": "t4",
            "_id": "AU8C4hcnDeuEUel6ntPr",
            "_score": 1,
            "_source": {
               "time": "2100-01-01"
            }
         },
         {
            "_index": "so",
            "_type": "t4",
            "_id": "AU8C4hr5DeuEUel6ntPs",
            "_score": 1,
            "_source": {}
         }
      ]
   }
}

The should logic makes literally "data should be in range or it should be missing"

Answer 2

Here is the NEST syntax I ended up with

var results = client
    .Search<ElasticResult>(s => s
    .Query(q => q
        .Filtered(filtered => filtered
            .Query(t=>t.Term(p => p.ShortDescription, "somevalue"))
            .Filter(ff => ff.
                Bool(b=> b
                    .Should(n=>n
                        .Range(p => p.OnField(f => f.EndTimeUTC).GreaterOrEquals(DateTime.UtcNow)) 
                        || 
                        n.Missing(m=>m.EndTimeUTC)
                        )
                    )
                )
            )
        )
    );

Answer 3

In the meantime I also found another, less elegant solution. Tell ES to provide a default value for the field when its null:

var client = new ElasticClient(settings);
client.Map<ElasticLotResult>(m=>m
    .Properties(props => props
    .Date(s => s
    .Name(p => p.EndTimeUTC)
    .NullValue(DateTime.MinValue)
    ))
);

then query and check for that default null value:

.Query(q => q
    .Term(p => p.ShortDescription, "somevalue")
    && (q.Range(p => p.OnField(f => f.EndTimeUTC).GreaterOrEquals(DateTime.UtcNow)) || 
    q.Term(t => t.EndTimeUTC, DateTime.MinValue))