Run coverage on test with subprocess

Question

Is there a way to test and report coverage on executable Python scripts? With some initial research, I found some guiding information in the documentation of the coverage package but the doc was not sufficiently clear for me to get it working with my package requirements:

• I use Python 3.8 under both Windows and Linux.
• The Python installation runs in a virtual environment.
• The code should run on a CI, ie without any manual or system tweaking.

I did not get it working together with subprocess (or alternatively exec ). My problem is that the scripts are never executed. The method to run the scripts does not really matter to me. The coverage package reports 0% coverage on the file example:

Name                  Stmts   Miss  Cover   Missing
---------------------------------------------------
scripts\__init__.py       0      0   100%
scripts\example.py        2      2     0%   1-3
scripts\scripts_test.py  14      1    93%   23
---------------------------------------------------
TOTAL                    16      3    81%

I am also annoyed by the fact that each file is opened in a separate window during the call to subprocess under Windows when using the flag shell=True (the meaning seems unclear to me).

Code

Example of a file to be tested This file shall be located in a subfolder called scripts . Let's call it scripts/example.py :

import numpy as np

print('My hello world example: ' + str(np.arange(3)))

The main file running the coverage on the tests looks as follows:

import coverage
import unittest
from pathlib import Path

if __name__ == '__main__':
    root_path = Path(__file__).resolve().parent

    coverage.process_startup()
    scripts_cov = coverage.Coverage(
        source=['scripts'], config_file=root_path / '.coveragerc')

    scripts_cov.start()
    test_suite = unittest.TestLoader().discover(
        root_path / 'scripts', pattern='scripts_*.py', top_level_dir=root_path)

    test_suite.run(result=unittest.TestResult())
    scripts_cov.stop()

    cov_rep = scripts_cov.report(show_missing=True, file=open('scripts_cov.txt', 'w'))

The test file - call it scripts/scripts_test.py - searches and runs all the scripts (here only scripts/example.py ). Despite the fact that the contained code is not run I guess that the coverage package has difficulties processing it too:

import coverage
import unittest
from pathlib import Path
from subprocess import Popen, PIPE
# from subprocess import run, call


class TestScriptsAsSubprocess(unittest.TestCase):

    def test_scripts(self):
        # Run all research code
        scripts_folder = Path(Path(__name__).parent / 'scripts')
        for file in scripts_folder.glob('*.py'):
            file_name_and_path = str(file.absolute())

            # Source of trouble:
            print(*(Popen(file_name_and_path, stdout=PIPE, stderr=PIPE).communicate()))

            # Non-working alternatives:
            # run(file_name_and_path, shell=True)
            # call(file_name_and_path, shell=True)
            # exec(open(file_name_and_path).read())

Answer 1

When you call .Popen() , prepend it with your python call

python3 -m coverage run

It's also usually worth separating the .Popen() call and .communicate() to get the outputs(s) and check the .returncode

p = subprocess.Popen(
    "python3", "-m", "coverage", "run", file_name_and_path)
    stdout=PIPE, stderr=PIPE
)
out, err = p.communicate()
if p.returncode != 0:
    raise OSError(...)

You may find you also need to set the cwd field to ensure coverage appends

If you intend to handle other types of interpreted files (ie. shell scripts or perl), you might consider first checking what a file is with file and choosing a dedicated coverage tool for each, perhaps exporting to a common format to combine at the end

Answer 2

Alternatively, make the files both a library and executable with the __name__ dunder check and moving the logic to a function .. this will allow you to both run them directly and also use normal unit testing against them

#!/usr/bin/env python3

def main():
    "normal run logic"

if __name__ == "__main__":
    main()