Using coverage and codecov.io with both pytest unit tests and bespoke inline tests

Question

The codebase for a Python project of mine contains lots of tests. These tests are broad and capacious enough that I would call them “robust”. I would like to know exactly how robust they are, so I have set up the coverage.py tool , and a codecov.io account .

The project has two types of tests:

1. a suite of unit tests, which were written from the start to run with pytest – ie there are no legacy unittest -based tests, and pytest fixtures and hooks are heavily leveraged.
2. per-module inline test-function suites, written with a simple bespoke test runner . These all look something like this:

# -*- coding: utf-8 -*-
import sys # …etc

# «module code»

def test():
    
    from clu.testing.utils import inline
    
    @inline.precheck
    def show_some_initial_values():
        """ Precheck function description """
        # «pre-check code»
    
    @inline
    def test_one():
        """ Test one’s description """
        # «test code»
    
    @inline
    def test_two():
        """ Test two’s description """
        # «test code»
    
    @inline.diagnostic
    def show_some_final_values():
        """ Diagnostic function description """
        # «post-run diagnostic code»
    
    return inline.test(100) # runs test functions 100 times;
                            # prechecks and diagnostics run once

if __name__ == '__main__':
    sys.exit(test())

… and they output reports like so:

… The inline tests can be run on a per-module basis, right from the editor. They can also all be run en masse via a nox setup that collects and runs all modules that define inline tests .

So now, regarding coverage.py and codecov.io – it was extremely easy to integrate the pytest suite with these tools. I made a few tweaks to an off-the-shelf .coveragerc file, installed the pytest codecov.io plugin, and that was that – those tests report their coverage to codecov.io just fine.

My question is, how do I integrate coverage reporting for the inline tests?

• Is there a simple way to configure coverage.py to “understand” these test functions, or do the inline tests need to report their results to the coverage tool(s)?
• Can one set something like this up within the .coveragerc file, or is it more involved?
• And, are there any other tools that I should consider, either additionally or instead of what I'm currently using?

Answer 1

Coverage.py doesn't understand anything about tests. All it does is tell you what parts of your code were run by some program. Usually that program is a test runner, but coverage.py doesn't care.

If you run your tests now with python mytestrunner.py , change the command to coverage run mytestrunner.py , and you will get data.