How do I handle elasticsearch's index timing when doing integration testing with mocha

Question

I'm working on a node project that uses elasticsearch. My test chain is gulp,mocha,chai,sinon. I'm having a hard time with getting my integration tests to work consistently. I want to load sample data in the database then run some tests over it. I believe I'm having problems because the documents I'm loading aren't indexed by the time my tests that use them run.

I've worked around this by doing the following:

    before(function (done) {
        testData.simpleLoadData(100, 2000);
        setTimeout(function () {
            done();
        }, 5000);
    });

This works fine locally, it works occasionally on travis. When I up the timer to 10000 it generally works in both places.

Is there a way to this without resorting to setTimeouts in the test code? Manually dealing with timing makes me a little squeamish.

Is manually dealing with timeouts the best option I have or are there better ways?

Note: these are integration tests and I explicitly want to use the external dependencies. I have unit tests that don't rely on the database already.

Answer 1

There are two things here that can trip you up: shard allocation and refresh cycles.

The first one can happen when you create a new index. The Create Index api will return a 200 OK as soon as the master acknowledges the request and begins the creation process. But the actual shard allocation happens asynchronously in the background. And while it is quick, integration tests can sometimes execute before the index is fully up and running, causing an error when you try to index documents.

The simplest way to make this robust is by creating the index and then calling the Health API with wait_for_status=green (or yellow, depending if you use replicas). This call will block until the index is fully allocated.

The next issue relates to the near-realtime aspect of search. By default, Elasticsearch refreshes the search index every second. This can be too slow for integration tests, and your documents may be indexed but not searchable when your tests run.

To fix this, index all your documents, then call a Refresh API on the target index. Once this call returns, your documents will be "live" and searchable.