Coveralls code coverage in Rails is misleading

Question

We are currently using Coveralls for code coverage of our Rails projects. The coverage results it is giving us are really unreliable. I have on numerous occasions found classes which have not been spec'd, written specs for them and then watched coverage actually drop. This is because Coveralls only checks classes which are loaded up by your specs. So if there is no spec for a class it is excluded from coverage statistics. This is obviously not ideal. Is there a way to get around this behavior? I am trying to push for greater emphasis on testing in my team and it's pretty hard when this is creating a false sense of security.

Answer 1

I've seen two reasons why coverage statistics are unreliable in Coveralls (and the similar service Codecov):

• When last I used it, Coveralls had to be told that a test suite was parallelized. If given no warning, Coveralls would display partial results. To avoid this, you can tell Coveralls that your parallelized test suite has completed. See the docs for details, but, briefly:

  • set the environment variable COVERALLS_PARALLEL=true on your CI server
  • POST { "payload": { "build_num": 1234, "status": "done" } } to https://coveralls.io/webhook?repo_token=(your repo token) at the end of your build. (Some hosted CI services figure out the payload automatically.)

• If you run your test suite more than once to retry flaky tests, partial results from a retry run can overwrite almost-full results from the first run. The problem I saw was specific to a homemade retry setup, but, if you're doing something like that, think through whether it might confuse Coveralls.

Regarding completely untested classes missing from your coverage report, that's not specific to Coveralls. If you want to be sure that all classes are loaded, load them eagerly before running your tests . Untested classes and methods are often unused, so auditing your app with a dead code detector like debride can also be a good step on the road to better coverage.