How to programmatically tell NUnit to repeat a test?

Question

How to programmatically tell NUnit to repeat a test?

Background: I'm running NUnit from within my C# code, using a SimpleNameFilter and a RemoteTestRunner. My application reads a csv file, TestList.csv, that specifies what tests to run. Up to that point everything works ok.

Problem: The problem is when I put the same test name two times in my TestList file. In that case, my application correctly reads and loads the SimpleNameFilter with two instances of the test name. This filter is then passed to the RemoteTestRunner. Then, Nunit executes the test only once. It seems that when Nunit sees the second instance of a test it already ran, it ignores it.

How can I override such behavior? I'd like to have NUnit run the same test name two times or more as specified in my TestList.csv file.

Thank you,

Joe

Answer 1

http://www.nunit.org/index.php?p=testCase&r=2.5

TestCaseAttribute serves the dual purpose of marking a method with parameters as a test method and providing inline data to be used when invoking that method. Here is an example of a test being run three times, with three different sets of data:

[TestCase(12,3, Result=4)]
[TestCase(12,2, Result=6)]
[TestCase(12,4, Result=3)]
public int DivideTest(int n, int d)
{
  return( n / d );
}

Answer 2

Running an identical test twice should have the same result. An individual test can either pass or it can fail. If you have tests that work sometimes and fail another then it feels like the wrong thing is happening. Which is why NUnit doesn't support this out of the box. I imagine it would also cause problems in the reporting of the results of the test run, does it say that test X worked, or failed if both happened?

The closest thing you're going to get in Nunit is something like the TestCaseSource attribute (which you already seem to know about). You can use TestCaseSource to specify a method, which can in turn, read from a file. So, you could for example have a file "cases.txt" which looks like this:

Test1,1,2,3
Test2,wibble,wobble,wet
Test1,2,3,4

And then use this from your tests like so:

[Test]
[TestCaseSource("Test1Source")]
public void Test1(string a, string b, string c) {
}

[Test]
[TestCaseSource("Test2Source")]
public void Test2(string a, string b, string c) {
}

public IEnumerable Test1Source() {
    return GetCases("Test1");
}

public IEnumerable Test2Source() {
    return GetCases("Test2");
}

public IEnumerable GetCases(string testName) {
    var cases = new List<IEnumerable>();

    var lines = File.ReadAllLines(@"cases.txt").Where(x => x.StartsWith(testName));
    foreach (var line in lines) {
        var args = line.Split(',');
        var currentcase = new List<object>();
        for (var i = 1; i < args.Count(); i++) {
            currentcase.Add(args[i]);
        }
        cases.Add(currentcase.ToArray());
    }
    return cases;
}

This is obviously a very basic example, that results in Test1 being called twice and Test2 being called once, with the arguments from the text file. However, this is again only going to work if the arguments passed to the test are different, since nunit uses the arguments to create a unique test name, although you could work around this by having the test source generate a unique number for each method call and passing it to the test as an extra argument that the test simply ignores.

An alternative would be for you to run the nunit from a script that calls nunit over and over again for each line of the file, although I imagine this may cause you other issues when you're consolidating the reporting from the multiple runs.