Getting either “DLL load failed” or python-exception in demandimport, when using mercurial on trac, this time on Windows 2008 R2, 64 bit

Question

This question , asked in 2011, and on a different operating system (XP instead of WSRV2008R2) is someone having a very similar sounding problem, but the accepted solution is to go back to a now-ancient mercurial version 1.8. I don't think that is a good answer in 2012.

I'd like to know if there is a way to get Apache 2.2 and Trac 0.12, and Mercurial 2.2.2 working on windows, and all running under Apache. The problems I'm having seem to be Python programming-related, and related to Python's use, and Mercurial's use, of various zipped and unzipped module formats.

1. It seems that to get this all working, may require you to build mercurial from sources, on windows, or to use Mercurial's "pure python" version. I'd like to know what about Python, or what about Mercurial, leads to these problems which might need me to run "pure python". Perhaps one accepted answer is that Mercurial is a quirky beast, and under mod_wsgi, one should use the pure python version of mercurial only. The underlying Python and mercurial implementation reasons why, would be great to know, as well.

2. It seems that Mercurial's shady looking demand-loading feature is implicated, which is a source-code level feature in the mercurial sources, which is confusing me, and I'm hoping someone can explain it.

3. The use in mercurial installed versions of a "library.zip" seems also to be creating an at-runtime situation where mercurial will import when I run python.exe from the command line (command prompt in Windows, with environment and path set as it would be set, if I was inside apache), but where Mercurial's primary python units will not import, from inside apache, with mod_wsgi.

Here is the test script which I use which works fine, from a command prompt, and shows that this python 2.6 instance has a workable set of Mercurial stuff in site-packages :

from mercurial import ui, hg
path = 'D:/MyHgRepo'
repo = hg.repository(ui.ui(), path)
print path,"Repository info:"
print repr(repo), "object len: ",len(repo)

However, when running from within Trac which is hosted inside Apache , attempts to load the mercurial module fails, first with a strange demandimport failure, and when that is fixed, silently with no reason given in Trac (I'll have to debug in trac sourcecode to solve this last bit, perhaps). Demandimport.py is a module inside the site-packages/Mercurial folder. This exception traceback comes from the trac.log:

In both cases, it says Look in the Trac log for more information. -- the error message here seems to be the problem:

Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\loader.py", line 68, in _load_eggs
    entry.load(require=True)
  File "build/bdist.linux-i686/egg/pkg_resources.py", line 1954, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "C:\Program Files\BitNami Trac Stack\python\lib\site-packages\mercurial\demandimport.py", line 114, in _demandimport
    mod = _origimport(name, globals, locals)
  File "build\bdist.win32\egg\trac\mimeview\rst.py", line 155, in <module>
  File "C:\Program Files\BitNami Trac Stack\python\lib\site-packages\mercurial\demandimport.py", line 87, in __getattribute__
    return getattr(self._module, attr)
AttributeError: 'module' object has no attribute 'directives'

The above exception can be fixed, oddly enough by "commenting out" that whole demand import unit. Here's my null "pass" implementation:

# demandimport.py - disabled by warren.

'''
demandimport - disabled in code by warren.
'''

def enable():
    "does nothing"
    pass

def disable():
    "does nothing"
    pass 

The above hack clears up the above visible exception, leaving trac.log free of errors, and yet, still there is no Hg plugin visible inside Trac.

From googling-wildly around the internet, I have found that:

1. Mercurial installers for Windows that install into the \\Python2.6 system folder, ship a "Library.zip" that contains not only .pyc but also .pyd files. This appears to work for some people, and fail for some people, when using mod_wsgi under apache, instead of while using Python interactively, or via standalone http hg serve operation. I have seen the author of the Trac Mercurial Plugin suggest unpacking library.zip. This is my first question; What is going on with Library.zip and is it my problem.

2. Lots of people have a hard time getting Python 2.6 or 2.7, mod_wsgi, Trac, and Mercurial to all run from inside Apache, on Windows. The underlying problems they have appear to be far beyond a software-user and installer's ability to solve, and require some arcane, or at least rudimentary python programming knowledge to solve.

That is why I'm asking this question here, on a programming answers site. I want to understand Python, and its library and module and site-packages architecture, as used by the popular open source packages Mercurial, Trac, and others, so that I can understand, diagnose and debug the broken Python codebase of Mercurial 2.2, which does not function on my system, when used in the mod_wsgi/apache2.2 environment.

I'm using Python 2.6, and Trac 0.12, all of which were installed by the Bitnami Trac Stack installer. I'm using Mercurial 2.2 installed via the mercurial-2.2-for-x86-for-Python2.6 installer from the mercurial website.

Answer 1

This happens because the version of Mercurial installed with binaries including some CPython extensions, that could not be imported into the version of Python that is being used inside the web server WSGI or CGI or other Python-in-apache technique.

Solution:

A. Find and install the correct version of mercurial binaries (including the library.zip and CPython binary extensions used by Mercurial)

B. Go for a pure 100%-python (no CPython binary extensions) version of Mercurial, such as hackable mercurial:

https://www.mercurial-scm.org/wiki/HackableMercurial