How can I generate tests programmatically (dynamically) and run them over a list of arbitrary parameters using pytest?

Question

I have a project where I am testing the ability of an API to handle certain queries. I would like to generate unique test cases programmatically based on a JSON input file. So, what I have done so far is:

# Open the JSON file with all test data:
@pytest.fixture(scope="class")
def test_data(request):
    base_path = os.path.abspath(os.path.dirname(__file__))
    path = os.path.join(base_path, request.file_name)
    with open(path, "r") as f:
        test_data = json.load(f)
    return test_data

Next I use this data in my test class:

@pytest.mark.parametrize("test_data", ["filename"], indirect=True)
def use_all_testdata(self, test_data):
    for t_dict in test_data:
        # What do I do here?

I am able to access my json file line by line, but I do not know how I can use this to programmatically generate pytest tests. I would like to be able to run a hundred tests this way if I had to. I have seen this question answered for unittest but not for pytest.

Answer 1

Just read the JSON file on module level, then parametrize with the read data:

base_path = os.path.abspath(os.path.dirname(__file__))
path = os.path.join(base_path, "file.json")
with open(path, "r") as f:
    test_data = json.load(f)


@pytest.mark.parametrize("data", test_data, ids=repr)
def test_json(data):
    assert data is not None

If you have multiple files, read them in the loop, eg

files = ["file1.json", "file2.json"]
test_data = []
base_path = os.path.abspath(os.path.dirname(__file__))
for filename in files:
    path = os.path.join(base_path, filename)
    with open(path, "r") as f:
        test_data.extend(json.load(f))

You can also implement the pytest_generate_tests hook. The advantage of this approach is that you don't have to copy the parametrize decorator to each test, should you have many:

def pytest_generate_tests(metafunc):
    base_path = os.path.abspath(os.path.dirname(__file__))
    path = os.path.join(base_path, "file.json")
    with open(path, "r") as f:
        test_data = json.load(f)
    metafunc.parametrize("data", test_data, ids=repr)


def test_json(data):
    assert data is not None


def test_another_json(data):
    assert "foo" in data.keys()

Just use data as argument in the test and it will be parametrized by metafunc.

Example: a file.json that contains

[
    {"foo": "bar"},
    {"foo": "baz"},
    {"spam": "eggs"}
]

yields tests:

test_spam.py::test_json[{'foo': 'bar'}] PASSED
test_spam.py::test_json[{'foo': 'baz'}] PASSED
test_spam.py::test_json[{'spam': 'eggs'}] PASSED