Why won't Python package install from my personal PyPI server?

Question

I've created a personal PyPI "packages" server on a Debian 9/Nginx box so that I can make my server builds deterministic. I pin all my Python packages and need to ensure that the exact versions of my Python packages as well as their sub-dependencies are always available whenever I need to rebuild my e-commerce servers.

I populated this server with my required packages using the pip2pi package. But when I run the "pip install" command on a client server to install my packages, I'm getting the following error:

Looking in indexes: https://packages.example.com/simple
Collecting Django==1.8.4 (from -r requirements.txt (line 2))
Collecting django-extensions==1.5.7 (from -r requirements.txt (line 3))
  Could not find a version that satisfies the requirement django-extensions==1.5.7 (from -r requirements.txt (line 3)) (from versions: )
No matching distribution found for django-extensions==1.5.7 (from -r requirements.txt (line 3))

I have about 40 packages in my requirements file so this is just an example of what happens. The Django package will get installed but pip will fail with the django-extensions package.

If I run this command on any of my client servers, I get the error shown above:

pip install -r requirements.txt

The requirements file looks like this:

-i https://packages.example.com/simple
Django==1.8.4
django-extensions==1.5.7
(more packages)

Now on my package server the root package directory, /var/www/packages, has the following structure:

# /var/www/packages:
├── Django-1.8.4-py2.py3-none-any.whl
├── django_extensions-1.5.7-py2.py3-none-any.whl
├── simple
    ├── django
    │   ├── Django-1.8.4-py2.py3-none-any.whl -> ../../Django-1.8.4-py2.py3-none-any.whl
    │   └── index.html
    ├── django-extensions
    │   ├── django-extensions-1.5.7-py2.py3-none-any.whl -> ../../django_extensions-1.5.7-py2.py3-none-any.whl
    │   └── index.html
    ├── index.html

This directory structure was built and the packages installed using pip2pi's pip2tgz command:

pip2tgz /var/www/packages/ -r requirements.txt

Here is the requirements input file read by pip2tgz:

-i https://pypi.org/simple
django==1.8.4
django-extensions=1.5.7
(more packages)

Here is how the nginx root directory is set on the package server:

server {
    ...
    root /var/www/packages;
    ...
}

The firewall on my package server is set to allow pings, allow me to SSH in, and to all http and https connections.

I'm using Python 3.5.3 and pip 19.0.3 on my client servers.

I'm not sure what I'm doing wrong. The pip command looks correct but I'm wondering if the pip2pi package is setting up my packages directories correctly. If I change the index argument in my client's requirement file to the default, https://pypi.org/simple , all packages restore without errors.

UPDATE 1

If I try to install django-extensions from the command line instead of via a requirements file, I still get an error:

pip install --index-url=https://packages.example.com/simple/ django-extensions
Looking in indexes: https://packages.example.com/simple/
Collecting django-extensions
  Could not find a version that satisfies the requirement django-extensions (from versions: )
No matching distribution found for django-extensions

In examining my other packages, I do see a pattern here. The error occurs whenever I try to install a package in which the wheel file's name contains an underscore ("_") but the package directory's normalized name has a hypen ("-") in it per Python PEP 503.

For example, django-rq fails:

directory: simple/django-rq
file: django_rq-0.9.0-py2.py3-none-any.whl

On the other hand, django-redis-cache doesn't fail:

directory: simple/django-redis-cache
file: django-redis-cache-1.6.5.tar.gz

But one other difference is that the former is a wheel file while the latter is a tgz file so that may account for the difference too. I'll continue to investigate this.

UPDATE 2

Per Ares' suggestion, I ran pip install with the verbose option:

pip install --verbose --index-url=https://packages.example.com/simple/ django-rq

Here is the error:

Created temporary directory: /tmp/pip-ephem-wheel-cache-g_hx5tb1
Created temporary directory: /tmp/pip-req-tracker-1bt5psaw
Created requirements tracker '/tmp/pip-req-tracker-1bt5psaw'
Created temporary directory: /tmp/pip-install-dqvtv6ek
Looking in indexes: https://packages.example.com/simple/
Collecting django-rq
  1 location(s) to search for versions of django-rq:
  * https://packages.example.com/simple/django-rq/
  Getting page https://packages.example.com/simple/django-rq/
  Looking up "https://packages.example.com/simple/django-rq/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): packages.example.com:443
  https://packages.example.com:443 "GET /simple/django-rq/ HTTP/1.1" 304 0
  Analyzing links from page https://packages.example.com/simple/django-rq/
    Skipping link https://packages.example.com/simple/django-rq/django-rq-0.9.0-py2.py3-none-any.whl (from https://packages.example.com/simple/django-rq/); wrong project name (not django-rq)
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-1bt5psaw'
Exception information:
Traceback (most recent call last):
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 269, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 196, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/flaugher/pip3/venv/lib/python3.5/site-packages/pip/_internal/index.py", line 688, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for django-rq

The key appears to be line 15 where it says "Skipping link... wrong project name (not django-rq)." I'm not sure why it's skipping the link.

Answer 1

It looks like the pip2pi project you are using is buggy and not maintained recently, with several open issues regarding dots/dashes in distribution names:

Issue with packages with hyphen in the name #84

Fix improper binary filename normalization for packages with hyphens in the names #67

Changed dir2pi to use original file name in file link. #85

I recommend to check out devpi-server instead.

Answer 2

For my private package server, the only thing that worked was --extra-index-url:

--extra-index-url https://foo.redacted.com/
Django
my_other_package

This will check pypi first, then your private server. One thing that's helped with debugging it is to use --verbose to show what's happening with the connection.

Answer 3

It looks like pypi-server will best meet my needs. It succeeds in installing the packages that pip2pi failed on as I described above. It's not overly-complicated to set up and it can be run on a remote server via Nginx or Apache. I found this article Setting up a PyPI server very helpful (although it did have a couple of typos).