Is there a way to run unit tests sequentially with MSTests?

Question

I am working in an application that is mostly single-thread, single user. There are a few worker threads here and there, and they are only using thread safe objects and classes. The unit tests are actually testing those with multiple threads (explicitly created for the tests), and they test fine.

The VSTS unit tests fail when testing business objects and sub-systems that are not thread safe. It is okay for them not to be thread-safe, that's the way the application uses them.

But the 'one thread per TestMethod' approach of MS tests kills us. I've had to implement object locks in many unit test classes just to ensure that the tests are run one after the other (I don't really care about the order, but I can't have two test methods hitting the same object at the same time).

The code looks like this:

[TestClass]
public class TestSomeObject
{
   static object turnStile = new object();
...
   [TestMethod]
   public void T01_TestThis()
   {
      lock(turnStile)
      {
      .. actual test code
      }
   }

   [TestMethod]
   public void T02_TestThat()
   {
      lock(turnStile)
      {
      -- actual test code
      }
   }

}

Is there a better/more elegant way to make the test run sequentially?

Answer 1

Use an Ordered Test .

Test > New Test > Ordered Test

Answer 2

There is the notion of an "Ordered Test" in which you can list tests in sequence. It is more geared towards ensuring a certain sequential order, but I can't see how that would be possible if B doesn't wait for A to complete.

Apart from that, it is unfortunate that your tests interfere with each other. There are Setup / TearDown methods that can be used per test such that it may after all be possible to isolate the tests from each other.

Answer 3

you can Use Playlist

right click on the test method -> Add to playlist -> New playlist

you can then specify the execution order

Answer 4

You can specifically require a mutex for each test execution, either in the specific tests you want to serialize, or for all the tests in a class (whatever shares the same mutex string).

For an entire test class, you can use the TestInitialize and TestCleanup attributes like so:

private readonly Mutex testMutex = new Mutex(true, "MySpecificTestScenarioUniqueMutexString");

[TestInitialize]
public void Initialize()
{
    testMutex.WaitOne(TimeSpan.FromSeconds(1));
}

[TestCleanup]
public void Cleanup() {
    testMutex.ReleaseMutex();
}

To be clear this isn't a feature of tests, ANY locking structure should work. I'm using the system provided Mutexes in this case: https://msdn.microsoft.com/en-us/library/system.threading.mutex(v=vs.110).aspx

Answer 5

I finally used the ordered test method. It works well.

However, I had a hell of a time making it work with the NAnt build. Running only the ordered test list in the build requires using the /testmetadata and /testlist switches in the MSTest invocation block. The documentation on these is sketchy, to use a kind description. I google all over for examples of "MSTest /testmetadata /testlist" to no effect.

The trick is simple, however, and I feel compelled to give it back to the community, in case someone else bumps into the same issue.

1. Edit the test metadata file (with a .vsmdi extension), and add a new list to the list of tests (the first node in the tree on the left pane. Give it the name you want, for example 'SequentialTests'.
2. If you used a /testcontainer switch for the MSTest invocation, remove it.
3. Add a switch for MSTest -> /testmetadata:
4. Add a switch for MSTEst /testlist:SequentialTests (or whatever name you used)

Then MSTest runs only the tests listed in the test list you created.

If someone has a better method, I'd like to hear about it!

Answer 6

I used ordered tests, also configured them easily on jenkins just use command

MSTest /testcontainer:"orderedtestfilename.orderedtest" /resultsfile:"testresults.trx"