Automated Tests in Microsoft Test Manager “build directory of the test run either does not exist or access permission is required”

Question

We are trying to run automated tests using Microsoft Test Manager 2015.

We've almost reached our goal. This is what was done so far:

• We are able to queue the 'build definitions' and successfully run Coded UI tests on a remote server.
• We've created a Test Environment and it's in a 'ready' state
• Test plan has build definition in use
• Test cases have association to the Coded UI tests.

More info:

• We don't have a build and copy to drop off folder steps because we don't have Visual Studio installed on the agent server
• So far, we manually put the .dll that contains coded UI tests to the C:\\build folder on the agent server
• Test project resides in a separate solution from the actual project we are testing. and we want to keep it this way.

When we try to run a separate coded UI test that is associated with a test case using Microsoft Test Manager 2015, we get an error:

The build directory of the test run either does not exist or access permission is required

• I don't know how to specify the build directory . And we are not really building anything. My idea is just to run the tests from C:\\build folder. Side note: Create Build Drop option is missing from our TFS 2015 vNext build definitions
• I've included Copy and Publish Build Artifacts step to the definiton, but I think it's useless because we are not even building anything. All DLLs are already on the drive
• I've checked the permissions for the C:\\build folder. the account we are suing has full access to it.

Any help will be greatly appreciated

Update:

• I've addedd Copy and Publish Build Artifacts step to the definition and specified a network share as a Drop folder
• Controller Service is running under a domain account
• Drop Folder has full control given to Everyone , including the controller service domain account
• I've manually copied assemblies with Coded UI test to this drop folder
• Controller Service log contains the following:

TcmRunner : Build Directory \\\\bhc-appqa01\\TFSShare\\Build Definition 2\\

TcmRunner: Error starting the test run: The build Directory of the test run either does not exist or access permission is required

So I'm super confused now... The controller says that it can see Build Directory and the next log message says build directory does not exist.

Answer 1

I feel you have mixed up a lot of things here.

1. The screenshot you have here is for new Build definition ie the new 2015 way. These new builds are NOT supported in MTM automated testing workflows. Your build definition must be the older xaml-based builds.
2. For your scenario the automated testing workflows consume a build drop. So the build that you specify has to have a drop location that can be accessed by the test controller. The older xaml builds provided this.

Good news is that given the fact you are on 2015 you can use the new build workflow to run your automated tests from a test plan/suite

Answer 2

Seems this issue is caused by there are no Visual Studio installed on the build agent.

---

Microsoft Visual Studio is required to be installed on the build server for the following scenarios:

• To build any CPP test project, you must install Visual Studio Professional or later.
• To run unit tests, or coded UI tests, you must install Visual Studio Professional or later.

• To use data and diagnostic data adapters:

  Code coverage: Visual Studio Premium or later.** Test impact: Visual Studio Ultimate. IntelliTrace: Visual Studio Ultimate.

• To build any modern style apps on a build machine: Visual Studio Ultimate or Visual Studio Express for Windows 8 (Operating system on
  build server must be Windows 8).

• To compile and run tests for a project with a faked assembly: Visual Studio Ultimate

Please see the source link from MSDN: Run tests in your build process

PS: As long as you have MSDN there is no additional license for Visual Studio required on a Build Agent.

---

Update

Seems you still need to assign a build to your test plan that contains your Coded UI tests when your running it with MTM. More details, please refer the similar question: Run CodedUI from MTM 2013

Answer 3

Back in August 2015 Microsoft announced that they are not investing in MTM anymore (see this blog article Lab management evolution )

In FAQ part at then end you will find following statement:

But, you can safely assume that we will not invest on new features in MTM Lab Center, nor in enhancing the XAML build templates. We will also not bring in any of the above features such as integrations with newer platforms, integration with newer versions of SCVMM, or integration with the new Build and Release Management services into MTM Lab Center . Similarly, we will not integrate the current Lab environments from MTM into the new Build and Release Management services . We will probably not ship another major version of MTM Lab Center.