使用 pytest.hookimpl 将 pytest 测试函数返回值写入文件

Question

I am looking for a way to access the return value of a test function in order to include that value in a test report file (similar to http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures ).

Code example that I would like to use:

# modified example code from http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures

import pytest
import os.path

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    if rep.when == "call" and rep.passed:
        mode = "a" if os.path.exists("return_values") else "w"
        with open("return_values.txt", mode) as f:
            # THE FOLLOWING LINE IS THE ONE I CANNOT FIGURE OUT
            # HOW DO I ACCESS THE TEST FUNCTION RETURN VALUE?
            return_value = item.return_value
            f.write(rep.nodeid + ' returned ' + str(return_value) + "\n")

I expect the return value to be written to the file "return_values.txt". Instead, I get an AttributeError.

Background (in case you can recommend a totally different approach):

I have a Python library for data analysis on a given problem. I have a standard set of test data which I routinely run my analysis to produce various "benchmark" metrics on the quality of the analysis algorithms on. For example, one such metric is the trace of a normalized confusion matrix produced by the analysis code (which I would like to be as close to 1 as possible). Another metric is the CPU time to produce an analysis result.

I am looking for a nice way to include these benchmark results into a CI framework (currently Jenkins), such that it becomes easy to see whether a commit improves or degrades the analysis performance. Since I am already running pytest in the CI sequence, and since I would like to use various features of pytest for my benchmarks (fixtures, marks, skipping, cleanup) I thought about simply adding a post-processing hook in pytest (see http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures ) that collects test function run time and return values and reports them (or only those which are marked as benchmarks) into a file, which will be collected and archived as a test artifact by my CI framework.

I am open to other ways to solve this problem, but my google search conclusion is that pytest is the framework closest to already providing what I need.

Answer 1

Sharing the same problem, here is a different solution i came up with:

using the fixture record_property in the test:

def test_mytest(record_property):
    record_property("key", 42)

and then in conftest.py we can use the pytest_runtest_teardown hook :

#conftest.py
def pytest_runtest_teardown(item, nextitem):
    results = dict(item.user_properties)
    if not results:
        return
    with open(f'{item.name}_return_values.txt','a') as f:
        for key, value in results.items():
            f.write(f'{key} = {value}\n')

and then the content of test_mytest_return_values.txt :

key = 42

Two important notes:

1. This code will be executed even if the test failed. I couldn't find a way to get the outcome of the test.
2. This can be combined with heofling 's answer using results = dict(item.user_properties) to obtain the keys and values that were added in the test instead of adding a dict to config and then access it in the test.

Answer 2

pytest ignores test functions return value, as can be seen in the code :

@hookimpl(trylast=True)
def pytest_pyfunc_call(pyfuncitem):
    testfunction = pyfuncitem.obj
    ...
    testfunction(**testargs)
    return True

You can, however, store anything you need in the test function; I usually use the config object for that. Example: put the following snippet in your conftest.py :

import pathlib
import pytest


def pytest_configure(config):
    # create the dict to store custom data
    config._test_results = dict()


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    if rep.when == "call" and rep.passed:
        # get the custom data
        return_value = item.config._test_results.get(item.nodeid, None)
        # write to file
        report = pathlib.Path('return_values.txt')
        with report.open('a') as f:
            f.write(rep.nodeid + ' returned ' + str(return_value) + "\n")

Now store the data in tests:

def test_fizz(request):
    request.config._test_results[request.node.nodeid] = 'mydata'