What is the difference between Python's 'Extras' and 'site-packages' directories?

Question

I'm confused about the way Python, on OS X, uses the packages in 'Extras' and 'site-packages'. In particular, I'm confused about the what I see in these directories, and how duplicate packages in the directories affect each other, and where I should put packages that I install.

I had assumed that 'Extras' ¹ was the location for packages that are not part of core Python, but that were nonetheless distributed with a given platform. For example, OS X, is distributed with PyObjC , twisted , and numpy , among others; and these are found in 'Extras'. I had also assumed that 'site-packages' ² is where packages that I subsequently install go, and that the directory is empty or absent on a "fresh" machine. Furthermore I'd assumed that installing a new package would leave the version in 'Extras' alone, and place any updates I do in 'site-packages', where they would "mask" those in 'Extras'. (So for example, the default installation could simply be restored by deleting 'site-packages'.)

What confuses me is that the contents of my 'Extras' directory is dwindling. Compared with the contents on a fresh OS X (10.8.2) install, the 'Extras' for my current configuration is missing

altgraph
altgraph-0.9-py2.7.egg-info
dateutil
macholib
macholib-1.4.2-py2.7.egg-info
modulegraph
modulegraph-0.9.1-py2.7.egg-info
numpy #though numpy-1.6.1-py2.7.egg-info is there
pkg_resources.py  # though .pyc is there
py2app
py2app-0.6.3-py2.7.egg-info
setuptools  # though setuptools-0.6c12dev_r88846-py2.7.egg-info is there
site.py  # though .pyc and .pyo are there
xattr
xattr-0.6.2-py2.7.egg-info
zope
zope.interface-3.5.1-py2.7.egg-info

and has two additional files not found in the fresh install

pkg_resources.py.OLD.1356069438.31
setuptools-0.6c12dev_r88846-py2.7.egg-info.OLD.1356069438.31
setuptools.OLD.1356069438.31

To the best of my recollection, these are all packages that I've updated myself, and versions of all are present in 'site-packages'.

Is the contents of the 'Extras' directory supposed to "dwindle" in this way? Does updating a package that is in 'Extras' "move" it to 'site-packages', or should the 'Extras' versions be left alone and simply "masked" by those in 'site-packages'?

---

^{1: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
2: /Library/Python/2.7/site-packages/}

Answer 1

OSX (and Linux) does not seem to like to install extra python packages in the default installation directory of python distribution.

pip can search (like python itself does) for installed modules, and if you do an upgrade will de-install the old version and install the new version in the place you asked for, defaulting to site-packages. If you closely look at the output of pip it will tell you what it removed and what it installed.

So only if you tell pip to install to the Extras directory, or would have a pip patched to do so for OSX by default, that would work.

It is actually a good thing the older versions are removed, as otherwise the version of a module you import would be depending on your search path (sys.path) and that is kind of error prone.

Whether it is supposed to dwindle can be argued, but it is a normal result of the way you do your upgrades with pip.