Fail tests on write to System.out or .err

Question

I'd like my CI build to fail when the tests write something to System.out or System.err. Preferably I would like to have a list of tests which produced unwanted output.

I tried to use mavens surefire plugin with an added listener, but that does only part of the trick as you can see from the question JUnit RunListener is removed on fail()

Did someone else try something like this and are there other ways to achieve a cleaner console on CI builds?

Answer 1

You might have a look at this Maven plugin https://github.com/policeman-tools/forbidden-apis which check for forbidden API calls. You can define wich calls should be forbidden in the code.

edit Simple example to show exclusion of not permitted api calls. In the example the call of System.out.println should not be permitted in test classes with the exception of one test class.

pom.xml

<build>
    <plugins>
        <plugin>
            <groupId>de.thetaphi</groupId>
            <artifactId>forbiddenapis</artifactId>
            <version>2.0</version>
            <executions>
                <execution>
                    <goals>
                        <goal>check</goal>
                        <goal>testCheck</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <signaturesFiles>
                    <signaturesFile>${project.build.testOutputDirectory}/signatures.txt</signaturesFile>
                </signaturesFiles>
                <excludes>
                    <!-- specify the classes which should be excluded -->
                    <exclude>**/Permitted*</exclude>
                </excludes>
            </configuration>
        </plugin>
    </plugins>
</build>

The signatur file src\\test\\resources\\signatures.txt

java.io.PrintStream#println(java.lang.String)

Test classes

// the one which doesn't use System.out
package sub.optimal;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.junit.Assert;
import org.junit.Test;
public class NoSystemOutTest {
    @Test
    public void dummy() {
        Logger logger = Logger.getGlobal();
        logger.log(Level.INFO, "some info");
        Assert.assertTrue(true);
    }
}

.

// the one in which the use of System.out should be permitted
package sub.optimal;
import org.junit.Assert;
import org.junit.Test;
public class PermittedSystemOutTest {
    @Test
    public void dummy() {
        System.out.println("permitted usage");
        Assert.assertTrue(true);
    }
}

.

// the one in which the use of System.out is not permitted
package sub.optimal;
import org.junit.Assert;
import org.junit.Test;
public class NotPermittedSystemOutTest {
    @Test
    public void dummy() {
        System.out.println("not permitted usage");
        Assert.assertTrue(true);
    }
}

Running the check with mvn test-compile forbiddenapis:testCheck will only report the forbidden api usage in NotPermittedSystemOutTest .

[ERROR] Forbidden method invocation: java.io.PrintStream#println(java.lang.String)
[ERROR]   in sub.optimal.NotPermittedSystemOutTest (NotPermittedSystemOutTest.java:9)

Answer 2

您可以使用Checkstyle插件，以便带有不需要的代码的测试失败，并将在surfire报告中列为失败。

Answer 3

You can check whether a test writes to System.out by using the library System Rules

public class YourTest {
  @Rule
  public final SystemOutRule systemOutRule = new SystemOutRule().enableLog();

  @After
  public void assertNoTextHasBeenWrittenToSystemOut() {
    assertEquals("", systemOutRule.getLog())
  }

  ...
}

There same may be done with System.err by using the SystemErrRule .