TFS Test result entry “Web Test Manager” or other test execution options

Question

QUESTIONS:

1. Do you have any direct or indirect experience with Web Test Manager by Sela Software Labs? Positive or negative experience is fine. I'm just looking for some facts to base production decisions from.

2. How risky is it to install Sela "Web Test Manager" to our Production Server WTM is a TFS Web Access extension. It extends website capabilities to include editing test steps and running tests.

3. Any other alternatives to executing tests and logging test results in TFS that we should consider?

Scenario: I have 2-3 Developers starting to run test cases as early as this week. We have 3 MTM (Microsoft Test Manager) licenses we use for testing (2 testers, one dev with VS2010 Ultimate). Purchasing another two full copies of Microsoft Test Professional for each of our VS2010 Pro/Premium (not Ultimate) devs just for periodically running test cases and doing light test case editing is not reasonable. We do not need trace listeners for general test pass runs.

Option #1: Sela Software Labs (Sela Group, Sela International) developed Web Test Manager several months ago but there is very little product reviews or customer feedback publicly available.

• Sela WTM website: http://www.sela.co.il/alm/products_WTM.html
• Single review posted on Microsoft partner marketing site: http://pinpoint.microsoft.com/en-us/applications/web-test-manager-wtm-is-the-only-tool-that-enables-to-test-with-tfs-2010-directly-from-the-browser-now-for-the-first-time-you-can-manage-your-tests-12884914644

Option #2: Developers track their results in individual spreadsheets which Testers then re-enter using MTM. This is not appealing at all and introduces several tedious failure risks.

Other options?

Answer 1

I created an open source tool called the TFS Test Steps Editor . Originally it was developed to work around MTM's lack of ability to insert line breaks in test steps. I just released a new version that has the ability to publish test results for manual tests.

MTM is a pretty big pain to use: it's slow and buggy, and worst of all, it will sometimes lose data while attempting to publish a result. My tool saves all of your in-progress test execution to disk as you work, and you can export a .ZIP of your results for backup or to re-open on another machine and resume testing.

Answer 2

Are you talking about automated tests or manual tests? For doing automated testing, I believe you don't need additional VS licenses..

Fancy version: Set up a test machine with the Visual Studio Test Agent (and Build/Workflow agent if you need to do deployment), a TFS Test Controller, and trigger test runs from a dev machine. If you do it right, results get automatically published to TFS and attached to the correct TestCase workitems and TFS build objects. Check out Visual Studio Test and Lab Management for more info on that (definitely some extra overhead in building out the infrastructure, but it's really slick once you've made the investment). You should be able to trigger the Build-Deploy-Test workflows from any VS license that has TFS access, I think.

Less fancy, but still doable: Still install the Test Agent , but don't worry about actually wiring it up to a Test Controller. The Test Agent installation will at least give you MSTest and the ability to /publish the results of test execution up to TFS for reporting/result storage.

If you're looking to do manual test execution/reporting, I unfortunately don't have a lot of suggestions.. Most of my team's investment has been on automation, so I don't have a ton of experience working with the manual testing interfaces. :/

Answer 3

I'm testing WTM to use it in our company. We have common reasons mentioned by you. For these issues WTM seems to be a good and the only one option. You are right, unluckely there are almost no reviews. So why I want to share my experience with WTM here.

1. Installation is easy and quick. Got no problems.
2. There is not much documenatation stuff on the web page. It's a pity. Hope Sela would make it better in the future (s. VisualAssist X Extension as an example for good public page). At least there is a good pdf documentation in the WTM folder after installing it.
3. Technically the extension is very good. But there are still some limitations (listed in documentation) and enhancements to be done (ie filtering, test step editing for customized TestCase TFS Work Item)

I think, for now it is a good option for testers who don't really need all MTM features.