How do you properly integrate unit tests for file parsing with pytest?

Question

I'm trying to test file parsing with pytest. I have a directory tree that looks something like this for my project:

project
    project/
        cool_code.py
    setup.py
    setup.cfg
    test/
        test_read_files.py
        test_files/
            data_file1.txt
            data_file2.txt

My setup.py file looks something like this:

from setuptools import setup

setup(
    name           = 'project',
    description    = 'The coolest project ever!',
    setup_requires = ['pytest-runner'],
    tests_require  = ['pytest'],
    )

My setup.cfg file looks something like this:

[aliases]
test=pytest

I've written several unit tests with pytest to verify that files are properly read. They work fine when I run pytest from within the " test " directory. However, if I execute any of the following from my project directory, the tests fail because they cannot find data files in test_files:

>> py.test
>> python setup.py pytest

The test seems to be sensitive to the directory from which pytest is executed.

How can I get pytest unit tests to discover the files in "data_files" for parsing when I call it from either the test directory or the project root directory?

Answer 1

One solution is to define a rootdir fixture with the path to the test directory, and reference all data files relative to this. This can be done by creating a test/conftest.py (if not already created) with some code like this:

import os
import pytest

@pytest.fixture
def rootdir():
    return os.path.dirname(os.path.abspath(__file__))

Then use os.path.join in your tests to get absolute paths to test files:

import os

def test_read_favorite_color(rootdir):
    test_file = os.path.join(rootdir, 'test_files/favorite_color.csv')
    data = read_favorite_color(test_file)
    # ...

Answer 2

One solution is to try multiple paths to find the files.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from coolprogram import *
import os


def test_file_locations():
    """Possible locations where test data could be found."""

    return(['./test_files',
            './tests/test_files',
            ])


def find_file(filename):
    """ Searches for a data file to use in tests """

    for location in test_file_locations():
        filepath = os.path.join(location, filename)
        if os.path.exists(filepath):
            return(filepath)
    raise IOError('Could not find test file.')


def test_read_favorite_color():
    """ Test that favorite color is read properly """

    filename = 'favorite_color.csv'
    test_file = find_file(filename)

    data = read_favorite_color(test_file)
    assert(data['first_name'][1] == 'King')
    assert(data['last_name'][1] == 'Arthur')
    assert(data['correct_answers'][1] == 2)
    assert(data['cross_bridge'][1] == True)
    assert(data['favorite_color'][1] == 'green')

Answer 3

One way is to pass a dictionary of command name and custom command class to cmdclass argument of setup function.

Another way is like here , posted it here for quick reference.

pytest-runner will install itself on every invocation of setup.py. In some cases, this causes delays for invocations of setup.py that will never invoke pytest-runner. To help avoid this contingency, consider requiring pytest-runner only when pytest is invoked:

pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)

pytest_runner = ['pytest-runner'] if needs_pytest else []

# ...

setup(
    #...
    setup_requires=[
        #... (other setup requirements)
    ] + pytest_runner,
)

Make sure all the data you read in your test module is relative to the location of setup.py directory.

In OP's case data file path would be test/test_files/data_file1.txt , I made a project with same structure and read the data_file1.txt with some text in it and it works for me.