Automation engine is unable to playback the test because it is not able to interact with the desktop… but desktop not locked

Question

I have been pulling my hair out over this one for two days now.

I got a new and much faster testing box, and it has started failing consistently every time at the same place in code, and very similar locations in the test cycle.

I am running Visual Studio 2017, though that is not really relevant. The latest TFS server, I think it is 2017 as well. I have two test boxes with the test power shell agent, test gui agent, and test controller, installed. One of them is really old and slow, it works great other than having timeout issues. The newer one I am trying to migrate to is running windows 10.

The windows 10 test box is logged in as a domain user I use for testing, this user is a local admin and is the user used to install all the testing agents. I run UltraVNC to connect to my test boxes remotely so as to avoid the issues that RDP and TeamViewer cause for the test agents.

The code will run great for 5 for six iterations, but during one specific test it always errors out on three iterations. The iterations are usually around the same point in time, but not always using the same data.

The code in question that triggers the error is:

Mouse.Click(save_tab);

where save_tab is a UITestControl object, a hyperlink on a web page.

The error triggered is:

Message: Automation engine is unable to playback the test because it is not able to interact with the desktop. This could happen if the computer running the test is locked or it's remote session window is minimized.

StackTrace: at Microsoft.VisualStudio.TestTools.UITesting.UITestLogGenerator.ExceptionRecorder(Exception exception, Boolean rethrow) at Microsoft.VisualStudio.TestTools.UITesting.Playback.ThrowIfScreenLockedOrRemoteSessionMinimized() at Microsoft.VisualStudio.TestTools.UITesting.Playback.MapAndThrowException(Exception exception, IPlaybackContext context) at Microsoft.VisualStudio.TestTools.UITesting.Playback.MapAndThrowException(Exception exception, String actionName, UITestControl uiControl) at Microsoft.VisualStudio.TestTools.UITesting.UITestActionExecutorCore.Click(UITestControl control, MouseButtons button, ModifierKeys modifierKeys, Point relativeCoordinate) at Microsoft.VisualStudio.TestTools.UITesting.Mouse.ClickImplementation(UITestControl control, MouseButtons button, ModifierKeys modifierKeys, Point relativeCoordinate) at Microsoft.VisualStudio.TestTools.UITesting.Mouse.<>c__DisplayClass32_0.b__0() at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestMethodInvoker.InvokeM ethod[T](Func`1 function, UITestControl control, Boolean firePlaybackErrorEvent, Boolean logAsAction) at Microsoft.VisualStudio.TestTools.UITesting.Mouse.Click(UITestControl control, MouseButtons button, ModifierKeys modifierKeys, Point relativeCoordinate) at Microsoft.VisualStudio.TestTools.UITesting.Mouse.Click(UITestControl control)

I have used VNC to actually watch the test machine as it has triggered these errors. The screen saver never turns on ( it is disabled ), the screen never goes to sleep ( that is also disabled ). Instead the entire test hangs for a period of time before triggering the errors.

In the catch portion of my try-cache for this error I try and take a screenshot using the testing browser object and get this error:

Message: Parameter is not valid.

StackTrace: at System.Drawing.Graphics.GetHdc() at System.Drawing.Graphics.CopyFromScreen(Int32 sourceX, Int32 sourceY, Int32 destinationX, Int32 destinationY, Size blockRegionSize, CopyPixelOperation copyPixelOperation) at System.Drawing.Graphics.CopyFromScreen(Int32 sourceX, Int32 sourceY, Int32 destinationX, Int32 destinationY, Size blockRegionSize) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.CaptureImagePrivate() at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.b__173_0() at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestMethodInvoker.InvokeMethod[T](Func`1 function, UITestControl control, Boolean firePlaybackErrorEvent, Boolean logAsAction) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.CaptureImage() at Testing.Base_CodedUITest.CaptureAndSaveImage(UITestControl doc, String FileName)

I then try taking a screenshot using:

UITestControl.Desktop.CaptureImage();

resulting in this error:

Message: Parameter is not valid.

StackTrace: at System.Drawing.Bitmap..ctor(Int32 width, Int32 height, PixelFormat format) at System.Drawing.Bitmap..ctor(Int32 width, Int32 height) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.CaptureImagePrivate() at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.b__173_0() at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestMethodInvoker.InvokeMethod[T](Func`1 function, UITestControl control, Boolean firePlaybackErrorEvent, Boolean logAsAction) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.CaptureImage() at Testing.Base_CodedUITest.CaptureAndSaveImage(UITestControl doc, String FileName)

It seems clear that the test is not able to communicate with the computer for some reason. I am not sure why though. Once this error has been handled the test continues on repeating the exact same test successfully several times in a row before triggering the same error again.

I do have one other possible clue on this issue. When connecting to this test box running UltraVNC I will periodically get a message that the connection has been terminated by the server ( the test box ), but it will always re-connect afterwards. I do not have this issue with my older test box. I have tried running a constant ping test to this newer test box during one of these UltraVNC connection hiccups and did not see any reduction in speed or lost packets. The connection across our entire lan is less than 1ms.

Does anyone have any idea what is going on?

EDIT: Upon recommendation I started looking through the event log. One event stuck out to me as being odd from Source = VSTTExecution.

(vstest.console.exe, PID 5356, Thread 9) TestTypes{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b} could not be loaded because the TIP could not be instantiated for the following reason(s): An item with the same key has already been added.

Source = VSTTController had this one, but do not think it is an issue since the tests are actually running. I suspect the controller and agent software I have installed may just be old and no longer needed.

The Controller service could not connect to Test Management Server. Could not load file or assembly 'Microsoft.VisualStudio.Threading, Version=15.6.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

And there was this one, Source = VSTTAgentProcess:

(QTDCAgent32.exe, PID 10416, Thread 51) DataCollectionAgentProcess: Parent process exited. Parent Exit Code: -1

EDIT 2:

As I try and collect more data on this I am finding this error occurring about the same time as the other ones. I think it is just a deeper error. Unfortunately it is also more generic, so finding anything on Google is difficult. I am pretty sure at this point that there is a problem deep within the OS, IE, or the test engine. My hope is to find a work around. In a month I will be able to test on yet another piece of hardware as well.

This second error is the original error that I am trying to take a screenshot of. I am trying to click a button. A button that is very visible and available, there is no visible error. For some reason the CodedUI test engine doesn't seem to recognize that IE even exists. However, once this error is past, the test engine is able to pick the browser back up and continue on with the next page in the test.

error level 2

Message: The playback failed to find the control with the given search properties. Additional Details: TechnologyName: 'Web' ControlType: 'Document' TagName: 'BODY' Failed to find any control that matched the search condition ControlType='Document'

StackTrace: at Microsoft.VisualStudio.TestTools.UITesting.Playback.MapControlNotFoundException(COMException ex, IPlaybackContext context) at Microsoft.VisualStudio.TestTools.UITesting.Playback.MapAndThrowComException(COMException innerException, IPlaybackContext context) at Microsoft.VisualStudio.TestTools.UITesting.Playback.MapAndThrowException(Exception exception, IPlaybackContext context) at Microsoft.VisualStudio.TestTools.UITesting.Playback.MapAndThrowException(Exception exception, String queryId) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.FindFirstDescendant(String queryId, Int32 maxDepth, Int32& timeLeft) at Microsoft.VisualStudio.TestTools.UITesting.SearchHelper.GetUITestControlRecursive(Boolean useCache, Boolean alwaysSearch, ISearchArgument searchArg, IList 1 windowTitles, Int32& timeLeft) at Microsoft.VisualStudio.TestTools.UITesting.SearchHelper.GetElement(Boolean useCache, ISearchArgument searchArg) at Microsoft.VisualStudio.TestTools.UITesting.SearchHelper.Search(ISearchArgument searchArg) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.FindInternal() at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.FindControlIfNecessary() at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.WaitForControlReadyPrivate(Int32 millisecondsTimeout, Boolean doLogging) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.<>c__DisplayClass186_0.b__0() at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestMethodInvoker.InvokeMethod[T](Func 1 windowTitles, Int32& timeLeft) at Microsoft.VisualStudio.TestTools.UITesting.SearchHelper.GetElement(Boolean useCache, ISearchArgument searchArg) at Microsoft.VisualStudio.TestTools.UITesting.SearchHelper.Search(ISearchArgument searchArg) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.FindInternal() at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.FindControlIfNecessary() at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.WaitForControlReadyPrivate(Int32 millisecondsTimeout, Boolean doLogging) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.<>c__DisplayClass186_0.b__0() at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestMethodInvoker.InvokeMethod[T](Func 1 function, UITestControl control, Boolean firePlaybackErrorEvent, Boolean logAsAction) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.WaitForControlReady(Int32 millisecondsTimeout) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.WaitForControlReady() at Testing.Base_CodedUITest.NavigateToUrl(String url) in c:\\agent_work\\2\\s\\VerizonOTM\\Other\\trc_UITesting\\Tests\\UI\\Base_CodedUITest.cs:line 348

error level 3

Message: Error HRESULT E_FAIL has been returned from a call to a COM component.

StackTrace: at Microsoft.VisualStudio.TestTools.UITest.Playback.Engine.IScreenElement.FindAllDescendants(String bstrQueryId, Object& pvarResKeys, Int32 cResKeys, Int32 nMaxDepth) at Microsoft.VisualStudio.TestTools.UITest.Playback.ScreenElement.FindAllScreenElement(String queryId, Int32 depth, Boolean singleQueryId, Boolean throwException, Boolean resetSkipStep) at Microsoft.VisualStudio.TestTools.UITest.Playback.ScreenElement.FindScreenElement(String queryId, Int32 depth, Boolean resetSkipStep) at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.FindFirstDescendant(String queryId, Int32 maxDepth, Int32& timeLeft)

Answer 1

I finally solved this problem.

The test in question that kept triggering these errors was downloading a file. If the file took too long to be generated and presented for download then the test tried to move on and log it as a failure. However, because it had not actually failed yet (it was just taking a long time) the browser was still busy and unavailable for CodedUI to interact with.

During the download process the browser gave no indication that a file was being downloaded. It looked like it was available for use in screen shots, coded ui was detecting it as available, the DOM said it was ready to go. And there was no page change that I could look for because the page was not actually changing.

The way my test was structured, I was on one page of my site, I dynamically built the URL to download the file, and tried to navigate to that URL. This would result in one of two things happening: either the file would error out and the site would actually navigate to that URL, or the file would be presented for download.

So my solution ended up being to set a very long timeout (10 minutes) and just keep looping through checking to see if the URL was still the same and if a file had been presented for download yet. This allowed me to wait even for the files that were taking longer to download. And once the browser had actually finished with the download request it became available for CodedUI again and my tests were able to continue on.