xunit test Fact multiple times

Question

I have some methods that rely on some random calculations to make a suggestion and I need to run the Fact several times to make sure is ok.

I could include a for loop inside the fact i want to test but because there are several test where I want to do this I was lookup for a cleaner approach, something like this Repeat attribute in junit: http://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without-loops/

Can I easily implement something like this in xunit?

Answer 1

You'll have to create a new DataAttribute to tell xunit to run the same test multiple times.

Here's is a sample following the same idea of junit:

public class RepeatAttribute : DataAttribute
{
    private readonly int _count;

    public RepeatAttribute(int count)
    {
        if (count < 1)
        {
            throw new ArgumentOutOfRangeException(nameof(count), 
                  "Repeat count must be greater than 0.");
        }
        _count = count;
    }

    public override IEnumerable<object[]> GetData(MethodInfo testMethod)
    {
        return Enumerable.Repeat(new object[0], _count);
    }
}

With this code in place, you just need to change your Fact to Theory and use the Repeat like this:

[Theory]
[Repeat(10)]
public void MyTest()
{
    ...
}

Answer 2

Had the same requirement but the accepted answers code was not repeating the tests so I have adapted it to:

public sealed class RepeatAttribute : Xunit.Sdk.DataAttribute
{
    private readonly int count;

    public RepeatAttribute(int count)
    {
        if (count < 1)
        {
            throw new System.ArgumentOutOfRangeException(
                paramName: nameof(count),
                message: "Repeat count must be greater than 0."
                );
        }
        this.count = count;
    }

    public override System.Collections.Generic.IEnumerable<object[]> GetData(System.Reflection.MethodInfo testMethod)
    {
        foreach (var iterationNumber in Enumerable.Range(start: 1, count: this.count))
        {
            yield return new object[] { iterationNumber };
        }
    }
}

While on the previous example the Enumerable.Repeat was being used it would only run the test 1 time, somehow xUnit is not repeating the test. Probably something they have changed a while ago. By changing to a foreach loop we are able to repeat each test but we also provide the "iteration number". When using it on the test function you have to add a parameter to your test function and decorate it as a Theory as shown here:

[Theory(DisplayName = "It should work")]
[Repeat(10)]
public void It_should_work(int iterationNumber)
{
...
}

This works for xUnit 2.4.0.

I've created a NuGet package to use this in case anyone is interested.

Answer 3

I know this is an old thread, but here is a little trick you can apply if you need to repeat a test a small number of times:

First, create some dummy data:

public static IEnumerable<object[]> DummyTestData()
{
   yield return new object[] { 0 };
   yield return new object[] { 1 };
   yield return new object[] { 2 };
   yield return new object[] { 3 };
 }

Then use the dummy data in order to force your test to run for each vector. In that case, the same test will be called 4 times (but the dummy data is actually not being used):

private static int _counter = 0;

[Theory]
[MemberData(nameof(DummyTestData))]
public void SomeTest(int dummyParam)     // dummyParam is unused
{
    _counter+= 1;
    DoSomething();

    Assert.True(...);           
}    

I find this approach very useful and less cumbesome than creating a new attribute.

Of course, this is not a good solution if you need the number of repetitions to be parameterisable (although I am sure someone could suggest a way to make my solution parameterisable :-)).

Answer 4

Simplest approach for small number of iterations: Make it a Theory instead of a Fact . Insert one line of [InlineData] for each iteration.

using Xunit;

namespace MyUnitTests
{
    public class Class1
    {

        [Theory]
        [InlineData]
        [InlineData]
        public void TestThis()
        {
            // test code here
        }
    }
}

Tested with XUnit 2.4.1

Answer 5

Since MemberData also takes a parameter, simply pass the iteration number to create the "DummyTestData".

public static IEnumerable<object[]> DummyTestData(int n)
{
    foreach (var iterationNumber in Enumerable.Range(start: 1, count: n))
    {
        yield return new object[] { iterationNumber };
    }
}

Now you can pass your repeat times as a parameter.

[Theory]
[MemberData(nameof(DummyTestData), n)] // repeat n times
public void SomeTest(int repeat)

Answer 6

You can use ITestCaseOrderer to order tests. While ordering you can specify some test multiple times. Eg

using System.Collections.Generic;
using System.Linq;
using Xunit.Abstractions;
using Xunit.Sdk;

namespace XUnit.Project.Orderers {
    public class RepeatOrderer : ITestCaseOrderer {
        public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase {
            List<TTestCase> result = new();
            testCases.ToList().ForEach(testCase => {
                var repeat = (ReflectionAttributeInfo)testCase.TestMethod.Method.GetCustomAttributes(typeof(RepeatAttribute)).FirstOrDefault();
                if (repeat != null && repeat.Attribute is RepeatAttribute) {
                    var attr = repeat.Attribute as RepeatAttribute;
                    Enumerable.Range(0, attr.Count).ToList().ForEach((_) => { result.Add(testCase); });
                }
                else {
                    result.Add(testCase);
                }
            });

            return result;
        }
    }
}

Attribute

public class RepeatAttribute : Attribute {
    public int Count { get; init; }
    public RepeatAttribute(int count) {
        this.Count = count;
    }
}

Instruct test project to use orderer. In my example TestProject1 is assembly where ordere is located.

[assembly : TestCaseOrderer("XUnit.Project.Orderers.RepeatOrderer", "TestProject1")]

And use Repeate attribute

[Fact]
[Repeat(6)]
public void Test1() {

...
}