Catch Fake Unit Tests in Java

Question

I have tests which do not have asserts in my repo, though jacoco gives good coverage. Is there a way to detect tests like this, other than better code reviews ?

Answer 1

Use PMD. It has a standard rule for unitTests without any asserts.

Answer 2

If your intent is only to find trivially wrong tests, namely those that lack any assert statement, checkers like PMD might do the job. You will likely get some false positives, namely for tests where the actual test goal is just to ensure that the SUT does not throw an exception (a case mentioned in the comments by Peter Lawrey).

There are, however, many more problems in tests that are not as simple to find: An assertion could just be wrong ( assertTrue(isPrime(9)) ),or, the assertion may only address a part of the relevant aspects (like, when dealing with rational numbers, only checking the numerator but not the denominator).

For the detection of such quality problems there also exist approaches, like mutation testing, which can help to some extent. And, while writing the tests in the first place: Using a test first development process ensures that a test at least failed once.

However, a test suite may have additional problems that are not related to the set of tests that are executed, but to non-functional quality criteria: Test execution time, maintainability of the test suite, expressiveness of the diagnostic output in case of a test failure etc.

When you are interested in detecting these kinds of problems, I would argue reviews are unavoidable.