How do I use easy_install and buildout when pypi is down?

Question

I am using buildout to automatically download and setup the many dependencies of my Plone installation. buildout more or less uses easy_install to download and install a bunch of Python eggs. This usually works, but it doesn't work if any of the dependencies cannot be downloaded or if I want buildout to find an internal package not appropriate for pypi. How can I set up my own local version of pypi to automatically mirror the packages I need? Will it still depend on third-party servers for packages that use pypi for their metadata but not their code?

Answer 1

Here are instructions on how to setup your own PyPi mirror . The homepage of this project is here . There also seems to be a growing number of mirrors out there.

For instructions on how to setup your own package index, check out this blog post where one solution is explained at the end. Then you can also host your own internal packages in there. The advantage is also that the versions are fixed that way. (For a way to pin the versions directly in buildout, check out this post ).

If there is only metadata on PyPI and the archive is stored somewhere else you might of course copy that over to your index as well. If you just use a PyPI mirror I assume that you still need access to these servers.

Answer 2

You can also use a mirror. Put this in the "[global]" section of "~/.pip/pip.conf":

index-url = http://d.pypi.python.org/simple/

This is a recent feature as announced here .

Answer 3

This page shows how to use the alternate mirror mentioned in @moraes post, but for easy_install, buildout and virtualenv as well as pip:

http://jacobian.org/writing/when-pypi-goes-down/

Answer 4

In case of zc.buildout: use its local download caching features. There are mostly three things to cache:

• external extends, ie http://dist.plone.org/release/4.1.2/versions.cfg
• eggs from some distserver, ie pypi
• downloads from zc.recipe.cmmi or similar recipes using the download infrastructure provided by zc.buildout

For all three we need to tweak the global configuration and set a cache folder for the extends and one for eggs and other downloads.

In your home folder create a .buildout folder.

In this folder create the folders extends-cache and downloads

In .buildout create a file default.cfg with:

[buildout]  
extends-cache = /home/USERNAME/.buildout/extends-cache 
download-cache = /home/USERNAME/.buildout/downloads

so you have:

.buildout/
├── default.cfg
├── downloads
└── extends-cache

Thats it. Make sure to not override these two variables from default.cfg in your specific buildout. After first successful run of buildout subsequent runs are running in offline mode ./bin/buildout -o .

As a side effect buildout is much faster if used in offline mode, ie when no new downloads are expected but some configuration changed

Beside this it makes sense to run your own pypi-mirror. As another source of information you might be interested in the article I wrote some time ago about this topic: http://bluedynamics.com/articles/jens/setup-z3c.pypimirror

Answer 5

Configure index in buildout.cfg , eg

[buildout]
index = http://a.pypi.python.org/
find-links = 

More mirrors on : http://www.pypi-mirrors.org/

Answer 6

For the packages, that you need to install in your virtualenv usually through a requirements.txt (or may be individually), you have to override your pip.conf file, usually located at ~/.pip/pip.conf

In your pip.conf file:

[global]
index-url=https://pypi.python.org/simple/

[install]
trusted-host=pypi.python.org

Here you can provide the url of your own custom version of pypi if needed.

If you wish to also use another pypi server while creating virtualenv through easy_install you need to override pydistutils.cfg file usually located at ~/pydistutils.cfg

In pydistutils.cfg file:

[easy_install]
index-url=https://pypi.python.org/simple/

This would ensure to create your venv with the url of pypi mentioned in pydistutils.cfg . Here, we're telling easy_install to use https://pypi.python.org/simple/ for creating venvs .

Answer 7

自2010年中期以来，PyPI一直在镜像http://pypi.python.org/mirrors