防止setup.py test / pytest安装额外的依赖项

Question

In my CI I have a build step, which uses pip and is correctly set up using private indexes etc.

Then I have test step, which executes python setup.py test . In this particular case test is an alias to pytest .

Setuptools and PIP resolve package dependencies differently, which leads to test step attempting to re-install some of the packages. I would like to prevent that. Is there a way to do that either with Setuptools configuration (preferable) or with PyTest configuration?

Update:

By popular demand repos to exemplify the issue I'm having https://github.com/vartec/example_repo_setuptools_issue

Answer 1

Why not override test command like this:

from setuptools import setup
from setuptools.command.test import test


class CustomTest(test):
    def run(self):
        self.distribution.install_requires = []
        super().run()


setup(
    name = "test",
    version = "0.0.1",
    install_requires=['non-existing-package'],
    cmdclass={
        "test": CustomTest,
    },
)

Answer 2

A small modification in setup.py specifically applicable for test should do the job

import sys

import pkg_resources
from setuptools import setup, find_packages
from setuptools.dist import Distribution

install_requires = [
    'redis~=2.8.0',
    'example_submodule',
]

tests_require = [
    'pytest',
]

original_function = None

if sys.argv[1] == "test":
    working_set = pkg_resources.WorkingSet()
    new_reqs = set()
    for req in install_requires:
        try:
            sets = working_set.resolve(pkg_resources.parse_requirements(req))
        except Exception as ex:
            new_reqs.add(req)
    install_requires = new_reqs


setup(
    name='example_module',
    version='0.1.0',
    packages=find_packages(),
    install_requires=install_requires,
    tests_require=tests_require,
    setup_requires=['pytest-runner'],
    extras_require={
        'testing': tests_require,
    },
    dependency_links=[
        'git+ssh://git@github.com/vartec/example_repo_setuptools_issue_submodule.git#egg=example_submodule-0.1.0',
    ]
)

See if this is indeed what you were looking for, if not then provide your feedback.

Edit-1

If you really don't care about the install_requires in case of test then you can just do

if sys.argv[1] == "test":
   install_requires = []

Answer 3

So, as I understand, the problem is that setuptools installs the internal dependencies into the local folder ( .eggs if I remember correctly), not into the normal virtualenv that you use.

And you do not like how setuptools does that. Neither do I, especially when I need a local DevPI server to be used (setuptools ignores that). So I do this way:

setup(
    ...
    install_requires=['monotonic'],  # just for example
    extras_require={
        'test': ['pytest', 'pytest-timeout'],
        'docs': ['sphinx', 'alabaster'],
    },
)

When you need to test, assuming that you create and install virtualenv somehow:

pip install -e .[test]

Here, . is the current directory. -e mean editable mode (but can be omitted). [test] is a setuptools "extra" . You can declare multiple "extras", and instal them as pip install mylib[ext1,ext2] if needed.

Then you can run the tests either way:

pytest
python setup.py test

The latter one is only if the test command is already configured to run pytest (see pytest integration manual ).

The trick is that if setuptools can find the test-dependencies in the current environment (virtualenv, pyenv, system python, whatever else), it will not install them as the eggs, and will just use the installed versions.

Actually, you do not even need to declare tests_require= in that case, as the libraries are assumed to be installed into the virtualenv. If they were not, the test command will just fail.

Similarly, you can pip install .[docs] , and build your docs with sphinx-build ... command from the current virtualenv.

Note that the install_requires are installed always, no matter with extras you add. So the app/lib itself will be always fully functional, importable and introspectable.

Hopefully, this was the question (if I understood it correctly).

Answer 4

You may use requirements.txt file specified dependencies are not in PyPI instead of the dependency_links parameter of setup method in setup.py .

requirements.txt:

-e git+ssh://git@github.com/gentcys/example_repo_setuptools_issue_submodule.git#egg=example_submodule-0.1.0

-e .[testing]

setup.py:

from setuptools import setup, find_packages

install_requires = [
    'redis~=2.10.0',
    'example_submodule',
]

tests_require = [
    'pytest',
]


setup(
    name='example_module',
    version='0.1.0',
    packages=find_packages(),
    install_requires=install_requires,
    tests_require=tests_require,
    setup_requires=['pytest-runner'],
    extras_require={
        'testing': tests_require,
    },
)

I forked your example repo and made some changes https://github.com/gentcys/example_repo_setuptools_issue.git
And i create a submodule repo https://github.com/gentcys/example_repo_setuptools_issue_submodule.git