pip cannot install arbitrary modules

Question

for every deploy to our test servers we install a number of packages using

pip install -r requirements.txt --upgrade

In probably half the cases pip fails to install a package returning an error such as

Could not find a version that satisfies the requirement django-brotli==0.1.3 (from -r configuration/environments/dev/../requirements.txt (line 66)) (from versions: )
No matching distribution found for django-brotli==0.1.3 (from -r configuration/environments/dev/../requirements.txt (line 66))

Two observations:

• The packages pip cannot install do exist - the installation worked fine in previous and in later builds.
• The package that fails is arbitrary

We're running multiple builds in parallel in different venvs, so I assume it's sort of a race condition. I've already made sure that the environment variable TMPDIR is set to a different location for each of the build processes but the problem still exists.

Any idea where else the pip processes could interfere which each other?

I'm using pip 18.1 with python 3.5.3

Answer 1

Thanks to @hoefling the additional verbose messages seemed to indicate that the problem was the caching that pip does, even though I wasn't really able to work out what the exact problem is. It just looked like in some cases pip tried to find the version in the cache and just couldn't while a parallel process was able to find it.

Since I added the option pip --no-cache the problem didn't reoccur.