Correct timing/placement of XCTestExpectation fulfill method call in unit tests?

Question

Consider the following unit test:

- (void)testSample {
    XCTestExpectation *expectation = [self expectationWithDescription:@"Sample"];

    [self.manager loadAllSuccess:^{
        [expectation fulfill];

        NSArray *data = [self.manager dataForIndex:0];

        // Correct count of data is 10, not 44 - this should fail.
        XCTAssertEqual(44, data.count);
    } failure:^(NSError *error) {
        [expectation fulfill];

        XCTFail(@"Error encountered");
    }];

    [self waitForExpectationsWithTimeout:60 handler:nil];
}

I am encountering some issues with the known failure case. The test finishes successfully, despite the fact that there are only supposed to be 10 items in the data array.

If I put the [expectation fulfill] call at the bottom of the block, after the XCTAssertEqual(44, data.count) method call, the test works as expected and fails until I correct the value to be 10.

Is this a known issue? I've been unable to read documentation saying that I should be calling that at the very last minute...

Answer 1

According to http://nshipster.com/xctestcase/

Always call fulfill() at the end of the asynchronous callback—fulfilling the expectation earlier can set up a race condition where the run loop may exit before completing the test. If the test has more than one expectation, it will not pass unless each expectation executes fulfill() within the timeout specified in waitForExpectationsWithTimeout().

Just found this - not sure if it's 100% accurate though as I cannot see any other documentation stating this elsewhere.

Answer 2

Yes, you should only call fulfill when your expectations have been fulfilled. If you expect your assertions to have been executed before the test ends, you must only fulfill your expectation after the assertion has been executed.

As @Zach's answer states, referencing NSHipster , it is entirely possible that the test will stop executing before your success/failure blocks are finished, because the only thing the test is waiting for is for the expectations to be fulfilled. The success and failure blocks are executing without knowledge of the test. As soon as the expectation is fulfilled, test execution will stop and any further code may not be executed.

Your code fulfills the expectation in the case of success and failure, so it seems that this test will always pass, regardless of the outcome. I suggest that you limit the number of places in which an expectation can be fulfilled to a single place, this way you will know what is working when your test passes and what has broken when it fails, without needing to debug further. You should split this test into two, one where an expectation is fulfilled in the success block but not the failure block, and vice versa on the second test.