Azure Devops Visual Studio Tests task cannot discover Nunit tests with the TestCaseSource attribute

Question

I am trying to use the [TestCaseSource] attribute from Nunit with my automated tests run from Azure DevOps.

Example: this test is not discovered

public static object[] TestSource = new[]
{
    new[] { new MyType() },
    new[] { new MyType() },
    new[] { new MyType() }
};
[Test, TestCaseSource("TestSource")]
public void MyTestMethod(MyType value)
{
    ...
}

Unfortunately my test methods cannot be executed because of the " No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct. " error.

However when I remove the TestCaseSource attribute from the test methods the tests are able to be executed, meaning they can be found normally, which leads me to the problem that the TestCaseSource attribute is making my tests undiscoverable.

Example: this test is discovered

[Test]
public void MyTestMethod()
{
    ...
}

I believe I've setup something wrong in the release and that this should work since the [TestCaseSource] test methods do work fine when I run the tests from Visual Studios Test Explorer.

My DevOps release pipeline is pretty simple, I use an extract files task with an Archive file patterns like " $(System.DefaultWorkingDirectory)/SolutionName/drop/SolutionName.Tests.zip " and a destination folder of " $(System.DefaultWorkingDirectory)/tests "

then I have a Visual Studio Test task (2.) that selects test by Test run and search folder of " $(System.DefaultWorkingDirectory)/tests "

My best guess is that maybe I need to set the " Path to custom test adapters " field of the Visual Studio Test task to the NUnit3Adapter Nuget package in my tests project but I am unsure of where/how to get the path to it.

Do I need to change my pipeline to include the NUnit3Adapter, or is there another reason why my tests are undiscoverable? I have searched extensively for a reason and as far as I can find no one else is having this issue with [TestCaseSource] leading me to believe I have done something wrong.

Update:

After more searching and investigation I think I've found the root of the issue, but I don't know how to fix it.

When I associate my test methods through visual studio Test Explorer to the Azure DevOps Test cases for associated automation, The name of the test method is not set correctly.

Example: given this test method from before

public static object[] TestSource = new[]
{
    new[] { new MyType() },
    new[] { new MyType() },
    new[] { new MyType() }
};
[Test, TestCaseSource("TestSource")]
public void MyTestMethod(MyType value)
{
    ...
}

If I associate the MyTestMethod to a work item, the "Automated test name" field in DevOps will show "SolutionName.Tests.MyTestMethod" as the value.

you can see here

However I believe this is wrong and instead the name needs to be " SolutionName.Tests.MyTestMethod(MyType) "

I think this because at one point somehow I did get one of my tests associated like "SolutionName.Tests.MyTestMethod(MyType)" and it worked fine.

My question now is how do I get the "associate test case" function to include the (MyType) at the end of the test method name?

Answer 1

It seems I found an answer to my own question and after further trial and error I managed to get everything working.

Effectively the end result was, as my update specified the "Associate to test case" function of the Visual Studio Test Explorer did not fill in the "Automated test name" as I would expect.

Example: without parameters

"SolutionName.Tests.MyTestMethod"

However eventually I figured out you must first execute the test locally . Then while the test result has passed, associate to the intended test case.

Only after would the parameter part of the test method name show up in the Azure DevOps "Automated test name" field correctly.

Example: with parameters

"SolutionName.Tests.MyTestMethod(MyType)"

Once I did this all of my tests were discovered and executed correctly on my agent.

I am not sure if this is intentional behavior or not. None of my searching returned this as the intended user experience. Never the less it worked for me.

Note

Although this works, I don't see each individual test result for each iteration from the object[] in azure dev-ops. Either that is a bug or what I am attempting to do was not intended behavior. Never the less besides that point everything works for me.