Does pytest support “default” markers?

Question

I am using pytest to test python models for embedded systems. Features to be tested vary by platform. ( I'm using 'platform' in this context to mean an embedded system type, not an OS type).

The most straightforward way to organize my tests would be to allocate them to directories based on platform type.

/platform1
/platform2
/etc.

pytest /platform1

This quickly became hard to support as many features overlap across platforms. I've since moved my tests into a single directory, with tests for each functional area assigned to a single filename (test_functionalityA.py, for example). I then use pytest markers to indicate which tests within a file apply to a given platform.

@pytest.mark.all_platforms
def test_some_functionalityA1():
    ...

@pytest.mark.platform1
@pytest.mark.platform2
def test_some_functionlityA2():
    ...

While I would love to get 'conftest' to automatically detect the platform type and only run the appropriate tests, I've resigned myself to specifying which tests to run on the command line.

pytest -m "(platform1 or all_platforms)"

The Question: (finally!)

Is there a way to simplify things and have pytest run all unmarked tests by default and additionally all tests passed via '-m' on the command-line?

For example: pytest -m "platform1"

would run tests marked @pytest.mark.platform1 as well as all tests marked @pytest.mark.all_platforms or even all tests with no @pytest.mark at all?

Given the large amount of shared functionality, being able to drop the @pytest.mark.all_platforms line would be a big help.

Answer 1

Let's tackle the full problem. I think you can put a conftest.py file along with your tests and it will take care to skip all non-matching tests (non-marked tests will always match and thus never get skipped). Here i am using sys.platform but i am sure you have a different way to compute your platform value.

# content of conftest.py
#
import sys
import pytest

ALL = set("osx linux2 win32".split())

def pytest_runtest_setup(item):
    if isinstance(item, item.Function):
        plat = sys.platform
        if not hasattr(item.obj, plat):
            if ALL.intersection(set(item.obj.__dict__)):
                pytest.skip("cannot run on platform %s" %(plat))

With this you can mark your tests like this::

# content of test_plat.py

import pytest

@pytest.mark.osx
def test_if_apple_is_evil():
    pass

@pytest.mark.linux2
def test_if_linux_works():
    pass

@pytest.mark.win32
def test_if_win32_crashes():
    pass

def test_runs_everywhere_yay():
    pass

and if you run with::

$ py.test -rs

then you can run it and will see at least two test skipped and always at least one test executed::

then you will see two test skipped and two executed tests as expected::

$ py.test -rs # this option reports skip reasons
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.2.5.dev1
collecting ... collected 4 items

test_plat.py s.s.
========================= short test summary info ==========================
SKIP [2] /home/hpk/tmp/doc-exec-222/conftest.py:12: cannot run on platform linux2

=================== 2 passed, 2 skipped in 0.01 seconds ====================

Note that if you specify a platform via the marker-command line option like this::

$ py.test -m linux2
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.2.5.dev1
collecting ... collected 4 items

test_plat.py .

=================== 3 tests deselected by "-m 'linux2'" ====================
================== 1 passed, 3 deselected in 0.01 seconds ==================

then the unmarked-tests will not be run. It is thus a way to restrict the run to the specific tests.

Answer 2

Late to the party, but I just solved a similar problem by adding a default marker to all unmarked tests.

As a direct answer to the Question: you can have unmarked tests always run, and include marked test only as specified via the -m option, by adding the following to the conftest.py

def pytest_collection_modifyitems(items, config):
    # add `always_run` marker to all unmarked items
    for item in items:
        if not any(item.iter_markers()):
            item.add_marker("always_run")
    # Ensure the `always_run` marker is always selected for
    markexpr = config.getoption("markexpr", 'False')
    config.option.markexpr = f"always_run or ({markexpr})"