Moved from pip to poetry and now pytest-cov won't collect coverage data

Question

I recently started using poetry to manage project dependencies, rather than using requirements.txt and test-requirements.txt and pip .

Since making the change, I'm not able to get coverage tests to work correctly. In both cases, I'm using tox to drive the testing (and I have the tox-poetry extension installed).

My tox.ini currently looks like this:

[tox]
isolated_build = True
envlist = pep8,unit

[testenv]
whitelist_externals = poetry

[testenv:venv]
commands = {posargs}

[testenv:pep8]
commands =
    poetry run flake8 {posargs:symtool}

[testenv:unit]
commands =
    poetry run pytest --cov=symtool {posargs} tests/unit

Previously, it looked like this:

[tox]
envlist = pep8,unit

[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
deps = -r{toxinidir}/requirements.txt
    -r{toxinidir}/test-requirements.txt

[testenv:venv]
commands = {posargs}

[testenv:pep8]
commands =
    flake8 {posargs:symtool}

[testenv:unit]
commands =
    pytest --cov=symtool {posargs} tests/unit

Since making the change to poetry , when I run eg tox -e unit , I see:

unit run-test: commands[0] | poetry run pytest --cov=symtool tests/unit
===================================== test session starts =====================================
platform linux -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
cachedir: .tox/unit/.pytest_cache
rootdir: /home/lars/projects/symtool, configfile: tox.ini
plugins: cov-2.11.1
collected 14 items

tests/unit/test_disasm.py .....                                                         [ 35%]
tests/unit/test_symtool.py .........                                                    [100%]r
Coverage.py warning: No data was collected. (no-data-collected)

I'm trying to figure out that no-data-collected issue. According to pytest --help , the --cov arguments set the path or package name:

 --cov=[SOURCE]        Path or package name to measure during execution

The root of the repository (which is rootdir in the above output from tox ) looks like this:

asm
pyproject.toml
README.md
reference
symtool
tests
tox.ini

There's definitely a symtool directory there containing the package. And even if the tests were somehow not running in the project root directory, the symtool package is installed in the test environment, as evidenced by the fact that the unit tests are actually passing (all of which include some variant of import symtool ).

How do I get coverage to work again?

Answer 1

Turning my comments into an answer:

This is an old issue with pytest-cov and tox (first reported in issue #38 ). tox installs your project as a third-party package and pytest will import it from the site (eg from .tox/unit/lib/python3.X/site-packages if the job is named unit ), while --cov=symtool instructs pytest-cov to collect coverage over the symtool dir in project root.

One solution is to switch to the src layout:

├── pyproject.toml
├── README.md
├── reference
├── src
|   └── symtool
├── tests
└── tox.ini

In your pyproject.toml , you will need to point poetry to source dir:

[tool.poetry]
...
packages = [
    { include = "symtool", from = "src" },
]

Now src will prevent importing code from the source tree, so --cov=symtool will collect coverage over installed modules, this being the only option. For the rest of the dev process, you shouldn't get into trouble as poetry install installs the project in editable mode, so the rest should just work.

Another option is to skip package installation with tox and install in editable mode instead. Example snippet to place in tox.ini :

[testenv]
whitelist_externals =
    poetry
skip_install = true
commands_pre =
    poetry install
commands =
    poetry run pytest ...

This pretty much kills testing of the installed modules though (you stop testing explicitly whether your project can be installed in a blank venv, instead using what's in the source tree), so I'd go with the src layout option.

Answer 2

If you are using poetry, you probably like to use the locked dependencies for pytest, flake8, ... in tox as well. This can be achieved with tox-poetry-installer .

To make sure, your tests run against the build and installed package and not the local files, you must use the --import-mode flag for pytest and set it to importlib .

To measure the coverage with pytest-cov, you have to point to the {envsitepackagesdir} .

To put it all together, your tox.ini can look like this:

[tox]
isolated_build = true
requires =
    tox-poetry-installer[poetry] == 0.6.0
envlist = py39

[testenv]
locked_deps =
    pytest
    pytest-cov
commands = pytest --cov {envsitepackagesdir}/mypackage --import-mode=importlib

What is --import-mode doing?

To run the tests, pytest needs to import the test modules. Traditionally this is done by prepending the path to the root folder where the discovered test folder is located to sys.path . Usually the test folder and the package folder share the same root folder. So the side effect of prepending to sys.path is, that tests run against the package folder and not the installed package, because python find's this folder first.

Instead of prepending to sys.path , one can advice pytest to append the discovered root folder to it. Doing this python will first look at the site-packages folder when trying to import. So one can test against the installed package.

Manipulating sys.path is almost always a bad idea, as it can to lead unwanted side effects. The docs of pytest describes one:

Same as prepend, requires test module names to be unique when the test directory tree is not arranged in packages, because the modules will put in sys.modules after importing.

The third option for importlib , was introduced with pytest 6. This uses pythons build-in importlib to load a test module dynamically without manipulating sys.path . They plan do make this option the default one in a future release.