Elastic NEST returns inconsistent results

Question

I am trying my hardest to understand how NEST works, but I think I have something working.. partially. I've imported 10.000 documents into my local Elastic. All the timestamps are 3 days old. When I use Kibana to get results, I have a simply query: hostname:www.website.com and I've set the date to 1 month to now, so I get all the results. That particular query gets me 44 results, which I've also verified to be correct in the CSV file.

Now, when I try to write the same code for the NEST library, I get various results. I get 33, 34, 43, and 44 results. Probably other amounts as well. It mostly happens after I start my application, and usually never when I just call my code many times.

public async Task Test()
{
    var model = new Model() 
    {
        Domain = "www.website.com"
    };

    var timestamp = DateTimeOffset.UtcNow.AddMonths(-1).ToUnixTimeSeconds();

    var nodes = new[]
    {
        new Uri("http://localhost:9200"),
    };

    var pool = new StaticConnectionPool(nodes);
    var settings = new ConnectionSettings(pool);
    settings = settings.BasicAuthentication("username", "password");
    var client = new ElasticClient(settings);

    var documents = new List<IReadOnlyCollection<ApacheRequest>>();

    Time processTimePerScroll = "2s";
    var numberOfSlices = Environment.ProcessorCount;
    if (numberOfSlices > 3) numberOfSlices = 3;

    var scrollAllObservable = client.ScrollAll<ApacheRequest>(processTimePerScroll, numberOfSlices, sc => sc
        .MaxDegreeOfParallelism(numberOfSlices)
        .Search(s => s
            .Index("apache-requests")
            .Query(q => 
                        q.Range(r => r.GreaterThanOrEquals(timestamp)) &&
                        q.Term(t => t.Servername, model.Domain)
            )
            .Sort(so => so.Field(f => f.Field(fi => fi.Timestamp)))
        )
    );

    var waitHandle = new ManualResetEvent(false);

    var scrollAllObserver = new ScrollAllObserver<ApacheRequest>(
        response =>
        {
            documents = documents.Concat(response.SearchResponse.Documents).ToList();
        },
        e =>
        {
            waitHandle.Set();
            throw new Exception(e.Message);
        },
        () => waitHandle.Set()
    );


    scrollAllObservable.Subscribe(scrollAllObserver);

    waitHandle.WaitOne();

    var requests = documents;

    var count = requests.Count(); //33, 34, 43, 44, and so on
}

When I test this against production data, I sometimes get 10k+ results back, so I need to use scroll.

What am I doing wrong here? I never throw the exception in the observer.

Answer 1

Seems like there is something wrong with that example I have. For one it's not async, but I knew that. I found a better example and adapted it to my needs, and it works flawlessly every time.

https://stackoverflow.com/a/56261657/2098652