XUnit output more info when test fails

Question

Does anyone know how to add additional info to the output when an XUnit test fails? I think it would be a useful thing to have. For example I'm processing test files, I'd like to see which one has failed.

This Question has information about using ITestOutputHelper, but that is only for printing info during successful running of the tests.

Answer 1

If you are having xUnit write the results to xml files, eg, by telling xUnit in the command line to write an XML results file, you can post-process the results from the XML files, to get more detail on failures, with a little C# code (or insert your favorite language in place of C#, eg, Python, Java, etc..

While the question seemed to be about which test case failed, for data driven tests, or runs of the test with a given input file, anything that gets written to the XML results file is fair game and available to this method. So if a failure exception is raised and you have the test case identifying info available to include in the exception, that should be captured in the XML results file.

Here is an example of some code for doing this in C#. A limitation of this code is that it only expects one assembly under the "assemblies" node in the XML.

public void ReadResultsXmlFile(string testResultsXmlFile)
{
    using (MyXmlTextReader = new XmlTextReader(testResultsXmlFile))
    {
        testResultXmlDocument = new XmlDocument();
        testResultXmlDocument.Load(MyXmlTextReader); // suppose that myXmlString contains "<Names>...</Names>"
        XmlNode xnAssembliesHeader = testResultXmlDocument.SelectSingleNode("/assemblies");
        TestRunDateTime = xnAssembliesHeader.Attributes.GetNamedItem("timestamp").Value;
        XmlNodeList xnAssemblyList = testResultXmlDocument.SelectNodes("/assemblies/assembly");
        foreach (XmlNode assembly in xnAssemblyList)
        {
            XmlNodeList xnTestList = testResultXmlDocument.SelectNodes(
                "/assemblies/assembly/collection/test");
            foreach (XmlNode test in xnTestList)
            {
                TestName = test.Attributes.GetNamedItem("name").Value;
                TestDuration = test.Attributes.GetNamedItem("time").Value;
                PassOrFail = test.Attributes.GetNamedItem("result").Value;
                // Failed test, get the failure information
                if (!(PassOrFail == "Pass"))
                {
                    Passed = false;
                    XmlNode failureData = test.SelectSingleNode("failure");
                    FailureText = failureData.InnerText;
                    StackTrace = failureData.SelectSingleNode("stack-trace").InnerText;
                    FailureMessage = failureData.SelectSingleNode("message").InnerText;
                    Console.WriteLine("Test: {0} Result: {1}  Error: {2}  StackTrace: {3}", TestName, PassOrFail, FailureMessage, FailureStackTrace);
                }
                else
                {
                    Passed = true;
                }
                Console.WriteLine("Test: {0} Result: {1}  Elapsed: {2}", TestName, PassOrFail, TestDuration);
            }
        }
    }
}

It will need approximately these declarations available, eg as part of the same class:

public class MyTestAssembly
{
    public string TestName { get; set; }
    public string PassOrFail { get; set; }
    public bool Passed { get; set; }
    public string TestOutcome { get; set; }
    public string AlertLevel { get; set; }  // None, Warning, Alert
    public string AssemblyName { get; set; }
    public string AssemblyPath { get; set; }
    public string TestRunDateTime { get; set; }
    public TimeSpan testDuration;
    public string TestDuration { get; set; }
    public DateTime TestStartTime { get; set; }
    public DateTime TestStopTime { get; set; }
    public string TestResultsXmlFile { get; set; }
    public string FailureMessage { get; set; }
    public string StackTrace { get; set; }
    public string FailureStackTrace { get; set; }
    //todo: Report an error if more than one assembly exists under assemblies node
    public XmlDocument testResultXmlDocument;
    public string FailureText { get; set; }
    private XmlNodeList MyListOfTestAssembliesInResultsFile { get; set; }
    private XmlTextReader MyXmlTextReader;
    public double TestDurationSeconds;
    // And more stuff... this is only a partial example, extracted from working code.
}

The XUnit results files I am working with are from xUnit 2.4.x, .NET Framework, and look like this:

<?xml version="1.0" encoding="utf-8"?>
<assemblies timestamp="05/06/2019 19:06:38">
  <assembly name="C:\Program Files (x86)\RunXUnitTests\TestAssemblies\HRCafeORT.DLL" environment="64-bit .NET 4.0.30319.42000 [collection-per-class, parallel (4 threads)]" test-framework="xUnit.net 2.4.1.0" run-date="2019-05-06" run-time="19:06:30" config-file="C:\Program Files (x86)\RunXUnitTests\TestAssemblies\xunit.console.exe.Config" total="1" passed="0" failed="1" skipped="0" time="7.376" errors="0">
    <errors />
    <collection total="1" passed="0" failed="1" skipped="0" name="Test collection for HRCafeORT.HRCafeWebsiteORTFeature" time="6.814">
      <test name="Ensure Intranet Application HR Cafe is available and working properly." type="HRCafeORT.HRCafeWebsiteORTFeature" method="EnsureIntranetApplicationHRCafeIsAvailableAndWorkingProperly_" time="6.8143311" result="Fail">
        <traits>
          <trait name="FeatureTitle" value="HR Cafe Website ORT" />
          <trait name="Description" value="Ensure Intranet Application HR Cafe is available and working properly." />
          <trait name="Category" value="HRCafe" />
          <trait name="Category" value="IE" />
        </traits>
        <failure exception-type="OpenQA.Selenium.ElementNotInteractableException">
          <message><![CDATA[OpenQA.Selenium.ElementNotInteractableException : Element cannot be interacted with via the keyboard because it is not focusable]]></message>
          <stack-trace><![CDATA[   at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
   at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.Remote.RemoteWebElement.Execute(String commandToExecute, Dictionary`2 parameters)
   at HRCafeORT.HRCafeWebsiteORTSteps.GivenIHaveSelectedTheLeaveSelection() in C:\AppTest\ORT\HRCafeORT\HRCafeWebsiteORTSteps.cs:line 49
   at TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
   at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
   at HRCafeORT.HRCafeWebsiteORTFeature.ScenarioCleanup()
   at HRCafeORT.HRCafeWebsiteORTFeature.EnsureIntranetApplicationHRCafeIsAvailableAndWorkingProperly_() in C:\AppTest\ORT\HRCafeORT\HRCafe.feature:line 14]]></stack-trace>
        </failure>
      </test>
    </collection>
  </assembly>
</assemblies>

A typical xUnitConsole command line, for the console test runner would look like below. The -xml followed by the output file name specifies the XML results file to be created during the run.

"C:\RunXUnitTests\TestAssemblies\xunit.console.exe" "C:\RunXUnitTests\TestAssemblies\HRCafeORT.DLL"  -xml "C:\Users\Public\Documents\TestResults\HRCafeORT.xml"

Answer 2

Failed tests throw exceptions. You can catch them and use ITestOutputHelper to output additional information.

public class AuditCheck
{        
    public AuditCheck(ITestOutputHelper output)
    {
        this.output = output;
    }

    [Fact]
    public void MyTest()
    {
        var expected = 23;
        var actual = 42;
        try
        {
            Assert.Equal(expected, actual);
        }
        catch (XunitException e)
        {
            output.WriteLine($"{e.Message}: My own output, like filename, etc. Today is {DateTime.Today.DayOfWeek} and i expected {expected} but got {actual}");
            throw;
        }
        Assert.Equal(expected, actual);
    }
}

The output is written to the log and aditionally you can view it by clicking on "Output" in Visual Studio Test Explorer ( Capturing Output ):