Installing PIP packages to a Virtualenv using a download cache

Question

What's the proper way to install pip packages to a virtualenv using cached packages? I've tried setting --timeout=360 and --use-mirrors , but pypi performance is so flakey, it routinely hangs or disconnects, making deployments a crapshoot.

So, my plan was to use the --download-cache option to pre-download all package archives (but not install them), eg:

pip install --upgrade --timeout=120 --use-mirrors --no-install --download-cache /usr/local/pip/cache -r pip-requirements.txt

and then specify this cache location during the install into the virtualenv, eg:

virtualenv /usr/local/myapp/.env
. /usr/local/myapp/.env/bin/activate; pip install --no-download --download-cache /usr/local/pip/cache -r pip-requirements.txt
deactivate

Creating the cache was very frustrating, because pip seems to nondeterministically save the downloaded package to the cache dir. Specifically, it refused to cache Django==1.4.0 , saying "that requirement has already been met", so I had to use the --force-reinstall .

Installing using the cache is currently where I'm stuck. Running the above command gives me the error:

Could not install requirement Django==1.4.0 (from -r pip-requirements.txt (line 1)) because source folder /usr/local/myapp/.env/build/Django does not exist (perhaps --no-download was used without first running an equivalent install with --no-install?)
Storing complete log in /home/chris/.pip/pip.log

What does this mean?

The files:

http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FD%2FDjango%2FDjango-1.4.tar.gz
http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FD%2FDjango%2FDjango-1.4.tar.gz.content-type

definitely exist in the cache directory, and I used the --no-install option.

What am I doing wrong?

Answer 1

The problem seems to be that --download-cache can only be used to specify the cache for downloading, not for installing. Therefore pip is still looking at /usr/local/myapp/.env/build/Django instead of /usr/local/pip/cache . Have you tried moving

pip install --upgrade --timeout=120 --use-mirrors --no-install --download-cache /usr/local/pip/cache -r pip-requirements.txt

to after the creation of virtualenv ? I wonder if that would help.

You may also want to try to export PIP_DOWNLOAD_CACHE and see if it works without using --download-cache .

Here is what I find that works:

pip install --no-install --use-mirrors --download=/DIRNAME PKGNAME
pip install --find-links=file:///DIRNAME --no-index --index-url=file:///dev/null PKGNAME

Now, actually , here is the tool for I would use instead of all the above and it solves all of the problems much more elegantly and reliably: pip2pi by David Wolever.

From the docs:

pip2pi builds a PyPI-compatible package repository from pip requirements

pip2pi allows you to create your own PyPI index by using two simple commands:

1. To mirror a package and all of its requirements, use pip2tgz :

    $ cd /tmp/; mkdir package/ $ pip2tgz packages/ httpie==0.2 ... $ ls packages/ Pygments-1.5.tar.gz httpie-0.2.0.tar.gz requests-0.14.0.tar.gz 

2. To build a package index from the previous directory:

    $ ls packages/ bar-0.8.tar.gz baz-0.3.tar.gz foo-1.2.tar.gz $ dir2pi packages/ $ find packages/ /httpie-0.2.0.tar.gz /Pygments-1.5.tar.gz /requests-0.14.0.tar.gz /simple /simple/httpie /simple/httpie/httpie-0.2.0.tar.gz /simple/Pygments /simple/Pygments/Pygments-1.5.tar.gz /simple/requests /simple/requests/requests-0.14.0.tar.gz 

3. To install from the index you built in step 2., you can simply use:

    pip install --index-url=file:///tmp/packages/simple/ httpie==0.2 

Bonus: you can even mirror your own index to a remote host with pip2pi .