如何在 pytest test_functions 之间重置 Python 运行时？

Question

. Answers to this question are eligible for a +50 reputation bounty. Dieshe wants to to this question.

I am using pytest to test code that creates a prometheus Python client to export metrics.

Between the test functions the prometheus client does not reset (because it has an internal state) which screws up my tests.

I am looking for a way to basically get a new Pyhon runtime before all test function calls. That way the internal state of the prometheus client would hopefully reset to a state that it had when the Python runtime started to execute my tests.

I already tried importlib.reload() but that does not work.

Answer 1

If you want to start each test with "clean" Prometheus client then I think best is to move it's creation and tear down to fixture with function scope (it's actually default scope), like this:

@pytest.fixture(scope=function)
def prometheus_client(arg1, arg2, etc...)
#create your client here
yield client
#remove your client here

Then you define your tests using this fixture:

def test_number_one(prometheus_client):
#test body

This way client is created from scratch in each test and deleted even if the test fails.

Answer 2

Straightforward approach: memorize current metric values before the test

First of all, let me show how I believe you should work with metrics in tests (and how I do it in my projects). Instead of doing resets, I keep track of the metric values before the test starts. After the test finishes, I collect the metrics again and analyze the diff between both values. Example of a django view test with a counter:

import copy
import prometheus_client
import pytest

from django.test import Client
from django.urls import reverse


def find_value(metrics, metric_name):
    return next((
        sample.value
        for metric in metrics
        for sample in metric.samples
        if sample.name == metric_name
    ), None)


@pytest.fixture
def metrics_before():
    yield copy.deepcopy(list(prometheus_client.REGISTRY.collect()))


@pytest.fixture
def client():
    return APIClient()


def test_counter_increases_by_one(client, metrics_before):
    # record the metric value before the HTTP client requests the view
    value_before = find_value(metrics_before, 'http_requests_total') or 0.0
    # do the request
    client.get('/my-view/')
    # collect the metric value again
    metrics_after = prometheus_client.REGISTRY.collect()
    value_after = find_value(metrics_after, 'http_requests_total')
    # the value should have increased by one
    assert value_after == value_before + 1.0

---

Now let's see what can be done with the registry itself. Note that this uses prometheus-client internals and is fragile by definition - use at your own risk!

Messing with prometheus-client internals: unregister all metrics

If you are sure your test code will invoke metrics registration from scratch, you can unregister all metrics from the registry before the test starts:

@pytest.fixture(autouse=True)
def clear_registry():
    collectors = tuple(prometheus_client.REGISTRY._collector_to_names.keys())
    for collector in collectors:
        prometheus_client.REGISTRY.unregister(collector)
    yield

Beware that this will only work if your test code invokes metrics registration again, Otherwise. you will effectively stop the metrics collection, For example, the built-in PlatformCollector will be gone until you explicitly register it again, eg by creating a new instance via prometheus_client.PlatformCollector() .

Messing with prometheus-client internals: reset (almost) all metrics

You can also reset the values of registered metrics:

@pytest.fixture(autouse=True)
def reset_registry():
    collectors = tuple(prometheus_client.REGISTRY._collector_to_names.keys())
    for collector in collectors:
        try:
            collector._metrics.clear()
            collector._metric_init()
        except AttributeError:
            pass  # built-in collectors don't inherit from MetricsWrapperBase
    yield

This will reinstantiate the values and metrics of all counters/gauges/histograms etc. The test from above could now be written as

def test_counter_increases_by_one(client):
    # do the request
    client.get('/my-view/')
    # collect the metric value
    metrics_after = prometheus_client.REGISTRY.collect()
    value_after = find_value(metrics_after, 'http_requests_total')
    # the value should be one
    assert value_after == 1.0

Of course, this will not reset any metrics of built-in collectors, like PlatformCollector since it scrapes the values only once at instantiation, or ProcessCollector because it doesn't store any values at all, instead reading them from OS anew.