Testing TestNG listeners

Question

I have a test A that programmatically runs another test B and test A passes only if test B fails. I want the build to succeed only when test A passes.

Precisely, I implemented TestNG listener ( MyListener ). I need to verify whether it is handling results of failing tests properly. The only approach to test it seems to have test A ( testListener ) run another test B ( AlwaysFailingTest ) programmatically and check the results.

@Test
public void testListener() {
    System.setProperty("knownBugs", "execute");
    tla = new TestListenerAdapter();
    TestNG testng = new TestNG();
    testng.setTestClasses(new Class[]{AlwaysFailingTest.class});
    testng.addListener(tla);
    testng.addListener(new MyListener());
    testng.run();
    assertThat(tla.getSkippedTests().size(), equalTo(0));
    assertThat(tla.getFailedTests().size(), equalTo(1));
    assertThat(tla.getPassedTests().size(), equalTo(0));
}

However, I don't want the test that is failing on purpose to impact test results of my library. Currently, for mvn clean test it is like that:

Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.407 sec <<< FAILURE!
shouldAlwaysFail(com.testing.listeners.MyListenerTest$AlwaysFailingTest)  Time elapsed: 0.01 sec  <<< FAILURE!
java.lang.AssertionError: Failing as always
        at org.testng.Assert.fail(Assert.java:94)
        at com.testing.listeners.MyListenerTest$AlwaysFailingTest.shouldAlwaysFail(MyListenerTest.java:99)

testListener() method is passing, that's great, but why results of the AlwaysFailingTest appear in the library test results, I don't understand. How can I avoid that?

I listed full code of MyListener and MyListenerTest here .

Answer 1

shouldAlwaysFail is being executed twice: once by the main test suite and once programatically by the testListener test. As it always fails the main test suite results will contain the failure. To fix this you can exclude the shouldAlwaysFail test from the main test suite so that it only gets run programmatically by the testListener test.

You can configure Maven Surefire to exclude your test:

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.19.1</version>
        <configuration>
          <excludes>
            <exclude>**/AlwaysFailingTest.java</exclude>
          </excludes>
        </configuration>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

Or configure TestNG to exclude your test:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="SuiteName" verbose="1">
    <test name="TestName">
        <packages>
            <package name="com.testing.listeners"/>
        </packages>
        <classes>
            <class name="com.testing.listeners.AlwaysFailingTest">
                <methods>
                    <exclude name="shouldAlwaysFail"/>
                </methods>
            </class>
        </classes>
    </test>
</suite>

See Maven Surefire Plugin – Inclusions and Exclusions of Tests and/or TestNG Documentation - 3 - testng.xml for more details.

Answer 2

Why don't you use "expectedExceptions" attribute of the @Test annotation so that the test will expect it to always FAIL and will not stop your execution.

@Test(expectedExceptions=AssertionError.class)
    public void test(){
        Assert.fail();
    }