Azure DevOps: tests failing when addressing multiple .Net frameworks

Question

[Edited: 10-Aug]

I have a project that generates a DLL (for a private NuGet package). This project is written to compile in the following frameworks:

<TargetFrameworks>netstandard2.0;netstandard2.1</TargetFrameworks>

The solution also contains four Test projects, each written in.NetCore 3.1 (consuming the.Netstandard 2.1 dll from the above project).

These has been working fine for months, but of course, I'm not testing that my NuGet package will work for code consuming the.NetStandard 2.0 framework.

I therefore updated the Test projects to compile in the following frameworks:

<TargetFrameworks>net472;net48;netcoreapp3.1</TargetFrameworks>

I immediately tripled my number of tests, and in Visual Studio they all pass (phew.), Well. almost tripled...each project contained a few tests that weren't applicable to the,NetFramework. so I used conditional compilation to remove these tests for the.NetFramework code.

All well and good....

However, when I push this up to Azure, then the test step fails.

The build script (yaml) contains the following commands:

...
- task: VSBuild@1
  displayName: 'Build all'
  inputs:
    solution: '$(solution)'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    maximumCpuCount: true
    
- task: DotNetCoreCLI@2
  displayName: 'Run Tests'
  inputs:
    command: 'test'
    projects: '**/*Test.csproj'
    arguments: '--configuration $(buildConfiguration) --collect "Code coverage"'
...

(fyi, we're also hooked into SonarCloud and report on code coverage)

The "headline" news seems promising:

Job preparation parameters
6 queue time variables used
100% tests passed

And when I dig into this I see the following pattern per test project:

Total tests: 3650
Passed: 3650

Total tests: 3650
Passed: 3650

Total tests: 3652
Passed: 3652

So, from the above, I can see from these numbers correspond to one of my test projects being executed three times, once for.Net 4.7.2, once for.Net 4.8 and once for.Core 3.1.

And I see a similar group of three executions for each of the other Test projects.

So...every test passed.

However, after the.NetCore version of each Test project executes (so, the one that had been executing fine for months), I now get the following:

##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1
"C:\Program Files\dotnet\dotnet.exe" test d:\a\1\s\My.Test.Project\EMy.Test.Project.csproj --logger trx --results-directory d:\a\_temp --configuration Release --collect "Code coverage"

I don't know how to interpret it, and thus how to fix it.

My guess is that the code analysis report is receiving the coverage data three times, which is causing it some problems. If that were the case then I'd have to say in the YAML "Execute each test project once using.NetCore 3.1 as we've been doing and collect the code coverage, but then execute them all again, once for.Net 4.7.2 and then 4.8 without the code coverage". But I'm not sure how to word that in yaml...

---

Edit 10th Aug #1

In response to Kevin Lu's comment below, I dug this out of the details.

Data collector 'Code Coverage' message: Failed to initialize code coverage datacollector with error: System.TypeLoadException: Could not load type 'Microsoft.VisualStudio.TestPlatform.Utilities.CodeCoverageRunSettingsProcessor' from assembly 'Microsoft.TestPlatform.Utilities, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. at Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollectorImpl.Initialize(XmlElement configurationElement, IDataCollectionSink dataSink, IDataCollectionLogger logger) at Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector.OnInitialize(XmlElement configurationElement). Data collector 'Code Coverage' message: Data collector 'Code Coverage' threw an exception during type loading, construction, or initialization: System.TypeLoadException: Could not load type 'Microsoft.VisualStudio.TestPlatform.Utilities.CodeCoverageRunSettingsProcessor' from assembly 'Microsoft.TestPlatform.Utilities, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. at Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollectorImpl.Initialize(XmlElement configurationElement, IDataCollectionSink dataSink, IDataCollectionLogger logger) at Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector.OnInitialize(XmlElement configurationElement) at Microsoft.VisualStudio.TraceCollector.BaseDataCollector.Initialize(XmlElement configurationElement, IDataCollectionEvents events, IDataCollectionSink dataSink, IDataCollectionLogger logger, IDataCollectionAgentContext agentContext) at Microsoft.VisualStudio.TraceCollector.BaseDataCollector.Initialize(XmlElement configurationElement, DataCollectionEvents events, DataCollectionSink dataSink, DataCollectionLogger logger, DataCollectionEnvironmentContext environmentContext) at Microsoft.VisualStudio.TestPlatform.Common.DataCollector.DataCollectorInformation.InitializeDataCollector() at Microsoft.VisualStudio.TestPlatform.Common.DataCollector.DataCollectionManager.LoadAndInitialize(DataCollectorSettings dataCol lectorSettings, String settingsXml)..

---

Edit 10th Aug #2

I tried adding different NuGet packages to see if I could get this to work i.e. Microsoft.TestPlatform.ObjectModel and then Microsoft.TestPlatform ), but to no avail.

I then changed the build script from:

- task: DotNetCoreCLI@2
  displayName: 'Run Tests'
  inputs:
    command: 'test'
    projects: '**/*Test.csproj'
    arguments: '--configuration $(buildConfiguration) --collect "Code coverage"'

to

- task: VSTest@2
  displayName: 'Run Tests'
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\No1Test.dll
      **\No2Test.dll
      **\No3Test.dll
      **\No4Test.dll
    searchFolder: '$(System.DefaultWorkingDirectory)'
    codeCoverageEnabled: true
    runInParallel: true

This felt like a retrograde step, however, all tests passed and no error messages so that's something to be pleased about. But....is this the "perfect" solution?

Answer 1

From the error message, there seems to be some problems with the package you are using.

I would like to share the packages in my project.

Project.csproj

...
  <PropertyGroup>
    <TargetFrameworks>net472;net48;netcoreapp3.1</TargetFrameworks>
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.6.1" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />
    <PackageReference Include="Microsoft.TestPlatform.ObjectModel" Version="16.6.1" />
  </ItemGroup>
...

The Microsoft.NET.Test.Sdk package relates with the code coverage. If I remove this package, the test will fail.

By the way, the build is running on Microsoft hosted agent: Windows-2019 .

Update:

Test with the Microsoft.NET.Test.Sdk version: 16.7.0. I get the same issue.

You can try the previous version(eg 16.6.1).

Update2:

Microsoft.NET.Test.Sdk version 16.7.1 was released that addresses this.