Installing python package from git at tag with setup.py

Question

I have a package (foo) in a private git repo. I want to install foo for use by another package, bar, via bar's setup.py. I want a specific version - the versioning in setup.py for foo matches its git tags (0.3.2, the git tag is v0.3.2)

The setup.py for bar looks like this:

#!/usr/bin/env python
  
from setuptools import setup, find_packages

setup(name='bar',
        install_requires=['foo@ git+ssh://git@github.com/fergusmac/foo.git@v0.3.2#subdirectory=somedir']
    )

I have also tried added the version explicitly at the end:

install_requires=['foo@ git+ssh://git@github.com/fergusmac/foo.git@v0.3.2#subdirectory=somedir==0.3.2']

I currently have version 0.3.1 installed in my venv. When I try to install this setup.py, via pip install . , or pip install . -U pip install . -U the version isn't upgraded - the repo isn't even checked out:

Requirement already satisfied, skipping upgrade: foo@ git+ssh://git@github.com/fergusmac/foo.git@v0.3.2#subdirectory=src==0.3.2 from 
git+ssh://****@github.com/fergusmac/foo.git@v0.3.2#subdirectory=src==0.3.2 in 
./venv/lib/python3.8/site-packages (from bar==0.0.0) (0.3.1)

However, when I use pip to install foo directly, the upgrade is done:

pip install git+ssh://git@github.com/fergusmac/foo.git@v0.3.2#subdirectory=src

Collecting git+ssh://****@github.com/fergusmac/foo.git@v0.3.2#subdirectory=src
  Cloning ssh://****@github.com/fergusmac/foo.git (to revision v0.3.2) to /tmp/pip-req-build-gxj2duq6
  Running command git clone -q 'ssh://****@github.com/fergusmac/foo.git' /tmp/pip-req-build-gxj2duq6
  Running command git checkout -q 40fa65eb75fc26541c90ee9e489ae6dd5538db1f
  Running command git submodule update --init --recursive -q
...
Installing collected packages: foo
  Attempting uninstall: foo
    Found existing installation: foo0.3.1
    Uninstalling foo-0.3.1:
      Successfully uninstalled foo-0.3.1
    Running setup.py install for foo... done
Successfully installed foo-0.3.2

I can't understand why installing with setup.py gives different behaviour. How do I ensure that it checks out the repo and looks for the correct version?

Follow up question - how would I specify 'check master branch for foo and install whatever version is there if it is higher than the current installed version'?

Answer 1

You're asking a precise and valid question, but I'm not confident that a satisfying answer will come. I'm not sure why what you're doing isn't working, but using direct URL dependencies with pip and setuptools is a new and rather complicated feature, and is possibly buggy/lacking on setuptools' side.

What I assume you want to do is have the package foo as a dependencies of bar - you don't actually need to use the PEP 508 direct URL specifier format. Instead you can provide either pip or setuptools with (relative) paths as dependency specifiers, and then use Git submodules to populate those paths. For example:

git submodule add git@github.com/fergusmac/foo.git
pip install ./foo

This will install whatever revision of foo was checked out when you added the submodule. As this answer explains, you can change the checked-out revision of the submodule and then install it like so:

cd foo
git checkout v0.3.2
cd ..
pip install ./foo

For setuptools you can specify it like this:

from pathlib import Path

...

setup(
    name='bar',
    install_requires=[
        f'foo @ file://localhost/{Path(__file__).parent}/foo/',
    ],
)

Path(__file__).parent is the directory containing bar's setup.py file. The path that follows that bit (eg /foo/ in this case) should be the location of foo's submodule relative to the directory containing bar's setup.py file.

---

Follow up question - how would I specify 'check master branch for foo and install whatever version is there if it is higher than the current installed version'?

Checkout master in the submodule, then install via pip install --upgrade . (assuming . is the project directory for bar).

---

See also: https://softwareengineering.stackexchange.com/a/365583/271937