How to parallelize a Data-Driven unit test in Visual Studio 2010?

Question

I know regular MS-Test unit tests can be parallelized on a multi-core machine (with caveats of course) by specifying parallelTestCount attribute in the .testresults file in the test solution. Like this,

<Execution parallelTestCount="1">
    <TestTypeSpecific />
    <AgentRule name="Execution Agents"></AgentRule>
</Execution>

More at: http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx

However, I have a data-driven test , something like this, this is just one test, but the input comes in from a csv and runs 1000s of records through the same test.

[DeploymentItem("InputDataRows.csv"), Timeout(37800000), DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\InputDataRow.csv", "InputDataRow#csv", DataAccessMethod.Sequential)]                
[TestMethod]
public void RunProcessing()
{
    int userId = Convert.ToInt32(TestContext.DataRow[0].ToString());
    int connId = Convert.ToInt32(TestContext.DataRow[1].ToString());
    string xml = TestHelper.GetDataFromDb(userId, connId);
    a = doStuffA(xml); 
    b = doStuffB(xml);
    Assert.IsTrue(a == b);
}

Because this is a slow process, I am looking at parallelizing this unit test.

The Sequential enum on the attribute is just the way it accesses data, the other option is Random, which is still serial and not parallel.

Answer 1

In order to parallelize this unit test, you'll need doStuffA() and doStuffB() to be able to operate on a subset of the data (eg a chunk or even a single row of your csv at a time). If you can refactor your methods to behave in this way, you can utilize tasks or a parallel foreach loop so that this test executes in parallel. Suppose your methods were refactored to handle a row of your csv, you could do something like this:

int userId = Convert.ToInt32(TestContext.DataRow[0].ToString());
int connId = Convert.ToInt32(TestContext.DataRow[1].ToString());
string xml = TestHelper.GetDataFromDb(userId, connId);
var rows = xml.Split('\n');

Parallel.ForEach(rows, (row) =>
{
    var a = doStuffOnRowA(row);
    var b = doStuffOnRowB(row);
    Assert.AreEqual(a, b);
});

Answer 2

This might seem a bit complex, but hear me out. There is a limitation in MSTest that you cannot actually run data-driven tests in parallel. What I have done in the past to get around this is to create a "custom tool" in visual studio.

https://msdn.microsoft.com/en-us/library/bb166508.aspx

OR

https://msdn.microsoft.com/en-us/library/bb166817.aspx

The custom tool that we created did the following:

1. Split out the csv into multiple csv files with only one row each.
2. Generate an individual test for each of the newly generated csvs.

When these tests were generated, we put specific test attributes on them, so we could specify to only run the tests with that attribute.

This sounds kind of over the top, but if you do a good job building the custom tool, it's actually a very smooth process.

Answer 3

If there is no data being changed in the xml (Or the methods are not modifying the same parts of the xml) then you could do..

var numCompleted = 0;
var a = Task.Run(() => { doStuffOnRowA(xml); });
var b = Task.Run(() => { doStuffOnRowB(xml); });
Task.WaitAll(new Task[2] { a, b });

This might not run if you copy and paste, kind of pseudo code. Not exactly Parallel, but will at least run around the same time!

Answer 4

Parallel execution of data driven tests is not supported. Please see here: RFC 004 - In-assembly Parallel Execution

Answer 5

As as I know: individual data rows in a test are NOT run in parallel. But if you have multiple unit tests, they do run in parallel.