如何获取在 Java 中 Cucumber 测试失败时抛出的异常？

Question

I can perform actions on test failure by using:

@After
public void afterTest(Scenario scenario) {
    if (scenario.isFailed()) {
        /*Do stuff*/
    }
}

However some of the actions I need to perform depend on the Exception that was thrown and in what context it was thrown. Is there a way to get the Throwable that caused the test to fail? For example in JUnit I would do this by extending TestWatcher and add a rule to my tests:

@Override
protected void failed(Throwable e, Description description) {
    /*Do stuff with e*/
}

However the cucumber-junit iplementation does not allow the use of rules, so this solution would not work with Cucumber.

I don't think I need to explain why accessing a thrown exception on test failure would be useful, however I will still provide an Example:

My test environment is not always stable, so my tests might fail unexpectedly at any moment (there's no specific place I can try to catch the exception since it could occur at any time). When this happens I need the test to reschedule for another attempt, and log the incident so that we can get some good statistical data on the environment instability (when, how frequent, how long etc.)

Answer 1

I've implemented this method using reflections. You can't access directly to steps errors (stack trace). I've created this static method which allows you to access to "stepResults" attribute and then you can iterate and get the error and do whatever you want.

import cucumber.runtime.ScenarioImpl; 
import gherkin.formatter.model.Result;  
import org.apache.commons.lang3.reflect.FieldUtils;  
import java.lang.reflect.Field;  
import java.util.ArrayList;

@After
public void afterScenario(Scenario scenario) {
  if (scenario.isFailed())
    logError(scenario);
}


private static void logError(Scenario scenario) {
   Field field = FieldUtils.getField(((ScenarioImpl) scenario).getClass(), "stepResults", true);
   field.setAccessible(true);
   try {
       ArrayList<Result> results = (ArrayList<Result>) field.get(scenario);
       for (Result result : results) {
           if (result.getError() != null)
               LOGGER.error("Error Scenario: {}", scenario.getId(), result.getError());
       }
   } catch (Exception e) {
       LOGGER.error("Error while logging error", e);
   }
}

Answer 2

The problem with the work around suggested by Frank Escobar:

By using reflection to reach into a frameworks internals you're depending on implementation details. This is a bad practice, when ever the framework changes its implementation your code may break as you will observe in Cucumber v5.0.0.

Hooks in Cucumber are designed to manipulate the test execution context before and after a scenario. They're not made to report on the test execution itself. Reporting is cross cutting concern and best managed by using the plugin system.

For example:

package com.example;

import io.cucumber.plugin.ConcurrentEventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;

public class MyTestListener implements ConcurrentEventListener {
    @Override
    public void setEventPublisher(EventPublisher publisher) {
        publisher.registerHandlerFor(TestCaseFinished.class, this::handleTestCaseFinished);
    }

    private void handleTestCaseFinished(TestCaseFinished event) {
        TestCase testCase = event.getTestCase();
        Result result = event.getResult();
        Status status = result.getStatus();
        Throwable error = result.getError();
        String scenarioName = testCase.getName();
        String id = "" + testCase.getUri() + testCase.getLine();
        System.out.println("Testcase " + id + " - " + status.name());
    }
}

When using JUnit 4 and TestNG you can activate this plugin using:

@CucumberOptions(plugin="com.example.MyTestListener")

With JUnit 5 you add it to junit-platform.properties :

cucumber.plugin=com.example.MyTestListener 

Or if you are using the CLI

--plugin com.example.MyTestListener 

Answer 3

You can to this by writing your own custom implementation of Formatter & Reporter interface. The empty implementation of Formatter is the NullFormatter.java which you can extend. You will need to provide implementations for the Reporter interface.

The methods which would be of interest will be the result() of the Reporter interface and possibly the done() method of Formatter. The result() has the Result object which has the exceptions.

You can look at RerunFormatter.java for clarity.

Github Formatter source

public void result(Result result) {
      //Code to create logs or store to a database etc...
      result.getError();
      result.getErrorMessage();
}

You will need to add this class(com.myimpl.CustomFormRep) to the plugin option.

plugin={"pretty", "html:report", "json:reports.json","rerun:target/rerun.txt",com.myimpl.CustomFormRep}

More details on custom formatters.

You can use the rerun plugin to get a list of failed scenarios to run again. Not sure about scheduling a run of failed tests, code to create a batch job or schedule one on your CI tool.

Answer 4

This is the workaround for cucumber-java version 4.8.0 using reflection.

import cucumber.api.Result;
import io.cucumber.core.api.Scenario;
import io.cucumber.core.logging.Logger;
import io.cucumber.core.logging.LoggerFactory;
import io.cucumber.java.After;
import org.apache.commons.lang3.ClassUtils;
import org.apache.commons.lang3.reflect.FieldUtils;

import java.io.IOException;
import java.lang.reflect.Field;
import java.net.URL;
import java.util.ArrayList;

@After
public void afterScenario(Scenario scenario) throws IOException {
    if(!scenario.getStatus().isOk(true)){
        logError(scenario);
    }
}

private static void logError(Scenario scenario) {
    try {
        Class clasz = ClassUtils.getClass("cucumber.runtime.java.JavaHookDefinition$ScenarioAdaptor");
        Field fieldScenario = FieldUtils.getField(clasz, "scenario", true);
        fieldScenario.setAccessible(true);
        Object objectScenario =  fieldScenario.get(scenario);

        Field fieldStepResults = objectScenario.getClass().getDeclaredField("stepResults");
        fieldStepResults.setAccessible(true);

        ArrayList<Result> results = (ArrayList<Result>) fieldStepResults.get(objectScenario);
        for (Result result : results) {
            if (result.getError() != null) {
                LOGGER.error(String.format("Error Scenario: %s", scenario.getId()), result.getError());
            }
        }
    } catch (Exception e) {
        LOGGER.error("Error while logging error", e);
    }
}

Answer 5

If you just want to massage the result being sent to the report then you can extend the CucumberJSONFormatter and override the result method like this:

public class CustomReporter extends CucumberJSONFormatter {

    CustomReporter(Appendable out) {
        super(out);
    }

    /**
     * Truncate the error in the output to the testresult.json file.
     * @param result the error result
    */
    @Override
    void result(Result result) {
        String errorMessage = null;
        if (result.error) {
            errorMessage = "Error: " + truncateError(result.error);
        }
        Result myResult = new Result(result.status, result.duration, errorMessage);
        // Log the truncated error to the JSON report
        super.result(myResult);
    }
}

Then set the plugin option to:

plugin = ["com.myimpl.CustomReporter:build/reports/json/testresult.json"]

Answer 6

For cucumber-js https://www.npmjs.com/package/cucumber/v/6.0.3

import { After } from 'cucumber'

After(async function(scenario: any) {
    const exception = scenario.result.exception
    if (exception) {
        this.logger.log({ level: 'error', message: '-----------StackTrace-----------' })
        this.logger.log({ level: 'error', message: exception.stack })
        this.logger.log({ level: 'error', message: '-----------End-StackTrace-----------' })
    }
})

Answer 7

After a lot of experimentation I now removed the Before/After-Annotations and rely on Cucumber-Events instead. They contain the TestCase (which is what the Scenario -class wraps) and a Result where you can call getError(); to get the Throwable.

Here is a simple example to get it working

import io.cucumber.plugin.EventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.Result;
import io.cucumber.plugin.event.Status;
import io.cucumber.plugin.event.TestCase;
import io.cucumber.plugin.event.TestCaseFinished;
import io.cucumber.plugin.event.TestCaseStarted;
import org.openqa.selenium.WebDriver;

public class TestCaseListener implements EventListener {
    @Override
    public void setEventPublisher(final EventPublisher publisher) {
        publisher.registerHandlerFor(TestCaseStarted.class, this::onTestCaseStarted);
        publisher.registerHandlerFor(TestCaseFinished.class, this::onTestCaseFinished);
    }

    public void onTestCaseStarted(TestCaseStarted event) {
        TestCase testCase = event.getTestCase();
        System.out.println("Starting " + testCase.getName());

        // Other stuff you did in your @Before-Method.
        // ...
    }

    private void onTestCaseFinished(final TestCaseFinished event) {
        TestCase testCase = event.getTestCase();
        System.out.println("Finished " + testCase.getName());

        Result result = event.getResult();
        if (result.getStatus() == Status.FAILED) {
            final Throwable error = result.getError();
            error.printStackTrace();
        }

        // Other stuff you did in your @After-Method.
        // ...
    }
}

All that's left to do is to register this class as a Cucumber-Plugin. I did this by modifying my @CucumberOptions -annotation:

@CucumberOptions(plugin = {"com.example.TestCaseListener"})

I find this much cleaner than all of this reflection-madness, however it requires a lot more code-changes.

Edit

I don't know why, but this caused a lot of tests to randomly fail in a multithreaded environment. I tried to figure it out, but now also use the ugly reflections mentioned in this thread:

public class SeleniumUtils {
private static final Logger log = LoggerFactory.getLogger(SeleniumUtils.class);

private static final Field field = FieldUtils.getField(Scenario.class, "delegate", true);
private static Method getError;

public static Throwable getError(Scenario scenario) {
    try {
        final TestCaseState testCase = (TestCaseState) field.get(scenario);
        if (getError == null) {
            getError = MethodUtils.getMatchingMethod(testCase.getClass(), "getError");
            getError.setAccessible(true);
        }
        return (Throwable) getError.invoke(testCase);
    } catch (Exception e) {
        log.warn("error receiving exception", e);
    }
    return null;
}
}