Python setup.py with private repository on GitLab as dependency_links based on commit ID

Question

I am trying to install a private dependency (not something that Python could find on PyPI).

I have added to the file setup.py this (as explained here: https://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi ):

dependency_links = [
        'https://gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID>'
      ]

On that official documentation they don't really explain in details what's the format of that URL, however using a <COMMIT_ID after the @ sounds reasonable (as it's done in a variety of other languages and dependency management tools).

When executing the command python setup.py install then I see in the logs/output this:

Reading https://gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID>

but then I don't see that package being actually installed as I can see from the logs/output for other dependencies.

I know there is a valid GitLab access token setup for my git command, because I've run this:

git config \
      --global \
      url."https://<ACCESS_TOKEN_NAME>:<ACCESS_TOKEN_VALUE>@gitlab.com".insteadOf \
      "https://gitlab.com"

and I can see it when checking the git config with:

git config --list | grep gitlab
url.https://<ACCESS_TOKEN_NAME>:<ACCESS_TOKEN_VALUE>@gitlab.com.insteadof=https://gitlab.com

• Is Python using the git command when running setup.py ?
• How can I specify a private GitLab dependency inside a Python setup.py file? It should be based on a commit ID rather than a package version
• What's wrong with the above?
• I have also the feeling this may be running differently when using pip install and targeting setup.py instead of running python setup.py install , is there a unique way of making this work with both flavors of Python installations? I am asking this because when fiddling with dependency_links I was trying various things like git+ssh instead of https and other variations, all of them failing to install that private repository with various log/outputs saying that repository was not found.

Edit

I've avoided dependency_links because it seems deprecated, so I used the solution proposed in the answer as:

install_requires=[
    ...
    "mylibraryname @ git+https://<ACCESS_TOKEN_NAME>:<ACCESS_TOKEN_VALUE>@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID>",
    ...
],

However when executing python setup.py install --record installed_files.txt , then the installation fails with this message:

Searching for mylibraryname@ git+https://<ACCESS_TOKEN_NAME>:<ACCESS_TOKEN_VALUE>@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID>
Reading https://pypi.org/simple/mylibraryname/
Couldn't find index page for 'mylibraryname' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for mylibraryname@ git+https://<ACCESS_TOKEN_NAME>:<ACCESS_TOKEN_VALUE>@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID>
error: Could not find suitable distribution for Requirement.parse('mylibraryname@ git+https://<ACCESS_TOKEN_NAME>:<ACCESS_TOKEN_VALUE>@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID>')

So I tried to use pip install . assuming there is a setup.py file in the current directory, this worked:

Collecting mylibraryname@ git+https://<ACCESS_TOKEN_NAME>:<ACCESS_TOKEN_VALUE>@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID> from git+https://<ACCESS_TOKEN_NAME>:****@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID> (from <MY_LIBRARY_WITH_SETUP_PY>==<MY_LIBRARY_VERSION>)
  Cloning https://<ACCESS_TOKEN_NAME>:****@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git (to revision <COMMIT_ID>) to /tmp/pip-install-bakazwe2/mylibraryname
  Running command git clone -q https://<ACCESS_TOKEN_NAME>:sYzRKNsYAnv5GtS6zLZj@gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git /tmp/pip-install-bakazwe2/mylibraryname

This solution seems to work only when using pip install . in a directory containing setup.py . This does not work with python setup.py install --record installed_files.txt .

Answer 1

https://python-packaging.readthedocs.io/ is quite old and outdated. Its sources was last updated at Dec 29, 2016 and most parts of it were not updated since 2012. Python packaging landscape changed significantly since that times. The new docs are at https://packaging.python.org/

dependency_links were declared obsolete and finally removed in pip 19.0. The replacement for it is install_requires with special syntax (supported since pip 19.1):

install_requires=[
    'package_name @ git+https://gitlab.com/<PRIVATE_ORG>/<PRIVATE_REPO>.git@<COMMIT_ID>'
]

See https://pip.readthedocs.io/en/stable/reference/pip_install/#requirement-specifiers and https://www.python.org/dev/peps/pep-0440/#direct-references

This requires pip install including pip install . and doesn't work with python setup.py install .

Answer 2

I've read multiple answers, but only this one worked for me (using pip 20.2.3 and Gitlab Pypi feature):

pip3 install --extra-index-url https://__token__:my_personal_token@gitlab.com/api/v4/projects/347/packages/pypi/simple .

My setup.py looked like:

from setuptools import setup

setup(name='whatever_production_scripts',
      version='0.0.1',
      description='Whatever production scripts',
      url='https://gitlab.com/user/whatever',
      author='Me Myself',
      author_email='user@whatever.com',
      license='All rights reserved',
      scripts=[
          'whatever_production_scripts/production/insomnia.py',
          'whatever_production_scripts/production/rdsmaintenance.py',
          'whatever_production_scripts/production/changeinstancetype.py',
      ],
      packages=[
          'whatever_production_scripts',
          'whatever_production_scripts.production',
      ],
      classifiers=[
          "Development Status :: 3 - Alpha",
          "Intended Audience :: System Administrators",
          "Operating System :: POSIX :: Linux",
          "Topic :: Internet",
          "Topic :: System :: Systems Administration",
          "Programming Language :: Python :: 3 :: Only"
      ],
      install_requires=[
          'privatepackage1>=0.1',
          'publicpackage1>=7',
          'publicpackage2>=2'
      ],
      zip_safe=False)