单元测试在调试时通过但在运行时失败

Question

A search method returns any matching Articles and the most recent Non-matching articles up to a specified number.

Prior to being returned, the IsMatch property of the matching articles is set to true as follows:

articles = matchingArticles.Select(c => { c.IsMatch = true; return c; }).ToList();

In a test of this method,

    [Test]
    public void SearchForArticle1Returns1MatchingArticleFirstInTheList()
    {
        using (var session = _sessionFactory.OpenSession())
        {
            var maxResults = 10;
            var searchPhrase = "Article1";
            IArticleRepository articleRepository = new ArticleRepository(session);
            var articles = articleRepository.GetSearchResultSet(searchPhrase, maxResults);
            Assert.AreEqual(10, articles.Count);
            Assert.AreEqual(1, articles.Where(a => a.Title.Contains(searchPhrase)).Count());
            var article = articles[0];
            Assert.IsTrue(article.Title.Contains(searchPhrase));
            Assert.IsTrue(article.IsMatch);
        }
    }

All assertions pass when the test is run in debug, however the final assertion fails when the test is run in release:

Expected: True But was: False

In the app itself the response is correct.

Any ideas as to why this is happening?

Edit:

I figured out what the problem is. It's essentially a race condition. When I am setting up the tests, I am dropping the db table, recreating it and populating it with the test data. Since the search relies on Full Text search, I am creating a text index on the relevant columns and setting it to auto populate. When this is run in debug, there appears to be sufficient time to populate the text index and the search query returns matches. When I run the test I don't think the index has been populated in time, no matches are returned and the test fails. It's similar to issues with datetimes. If I put a delay between creating the catalog and running the test the test passes.

Answer 1

Pones, you have since clarified that the unit test fails when not debugging.

At this stage it could be anything however you should continue to run the unit test not debugging and insert the following statement somewhere you know (or think you know) is true

 if(condition)
            Debugger.Launch();

This will do the obvious and allow you to zone in on whats going wrong. 1 Place i suggest is on the IsMatch property (for starters)

Another common place you can run into issues like this is using DateTime's. If your unit test is running 'too fast' then it may break an assumption you had.

Answer 2

try to print out the actual result that you are comparing them with expected on debug and normal run

in my case, I created entities (JBA) in the test method

in debug mode, the generated ids were 1, 2 and 3

but in the normal running mode, they ware different

that caused my hard-coded values to make the test fail, so I changed them to get id from entity instead of the hard-coded way

hope this helps

Answer 3

Obviously the problem will be different for other users, but I just hit it, and figured my solution may help some. Basically when you are running in debug mode, you are running a single test only. When you are running in run mode, you are running multiple tests in addition to the one you are having a problem with.

In my situation the problem was those other tests writing to a global list that I was not explicitly clearing in my test setup. I fixed the issue by clearing the list at the beginning of the test.

My advice to see if this is the type of problem you are facing would be to disable all other tests and only 'run' the test you have an issue with. If it works when ran by itself, but not with others, you'll know you have some dependency between tests.

Another tip is to use Console.WriteLine("test") statements in the test. That's actually how I found my list had items with it leftover from another test.