Not all Specflow Hooks are executed when running tests on vstest.console with NUnit3TestAdapter

Question

We use the visualstudio.com Release Manager to tell a local Build Agent to build and execute a set of SpecFlow tests on the Build Agent-machine, pulling from a GIT-branch.

We're using the Test Assemblies-task with a Custom Test Adapter (NUnit3TestAdapter 3.7.0) and point it toward the assembly containing the SpecFlow tests. Other assemblies inside the same bin folder contain SpecFlow-Hooks that execute setup and teardown-hooks.

The problem we're facing is that not all of these Hooks seem to called. The first and last messages in the .trx logs are all from [BeforeScenario] and [AfterScenario] hooks. There are -however- [BeforeTestRun] , [AfterTestRun] and [BeforeFeature] hooks in the same assemblies.

I've pasted the (abridged) logs from VSTS below:

Executing vstest.console.exe " DLL CONTAINING TESTS " /TestCaseFilter:"TestCategory=MP2" /Settings:"C:\\agent_work\\r17\\a\\TestResults\\1_2017-05-31_09-31-50-AM.runsettings" /logger:trx /TestAdapterPath:"C:\\agent_work\\r17\\a\\drop\\b\\packages\\NUnit3TestAdapter.3.7.0\\tools"

Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1 2017-05-31T07:31:51.1862622Z Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...

Information: NUnit Adapter 3.7.0.0: Test execution started

Information: Running all tests in C:\\agent_work\\r17\\a\\drop\\b\\ DLL CONTAINING TESTS

Information: NUnit3TestExecutor converted 472 of 472 NUnit test cases

Then the logging from the tests itself, starting with messages from BeforeScenario .

I'm at a loss as to why the other hooks are never called. Does NUnit3TestAdapter even keep track of a TestRun context? Is it because the Before/AfterTestRun and Before/AfterFeature-hooks are bound to static methods? Or is it possible they ARE executed, but the messages are never passed through to the .trx logs?

Answer 1

It turns out [BeforeTestRun] was being called perfectly fine, but just didn't show up in the logs. The problem was with [AfterTestRun] , which turns out to be an old bug in SpecFlow that was never really fixed (?)