Apache mod_wsgi崩溃

Question

I'm trying to set up ReviewBoard and I'm running into an issue where mod_wsgi seems to be crashing. I get an error message like this:

AH00052: child pid 44359 exit signal Segmentation fault (11) 

I tried to debug using GDB as stated here: https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB . I'm not having any luck. When I attach gdb and hit the URL that crashes, the only info I get is this:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f170ab51700 (LWP 45017)]
0x00007f17159f957f in __strlen_sse42 () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install expat-2.0.1-11.el6_2.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-37.el6_6.x86_64 libcom_err-1.42.8-1.0.2.el6.x86_64 libffi-3.0.5-3.2.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64 libuuid-2.17.2-12.18.0.1.el6.x86_64 openssl-1.0.1e-30.el6_6.8.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) 

I'm running Apache 2.4.12, mod_wsgi 4.4.11 and Python 2.6.6 on Oracle Enterprise Linux 6. I'm pretty stuck here, because this is beyond anything I've had to do before.

EDIT: Obtained some more info:

Thread 2 (Thread 0x7fc77b65a700 (LWP 46498)):
#0  __strlen_sse42 () at ../sysdeps/x86_64/multiarch/strlen-sse4.S:32
No locals.
#1  0x00007fc78302ce4b in PyString_FromString (str=0x0) at Objects/stringobject.c:116
        size = <value optimized out>
        op = <value optimized out>
#2  0x00007fc783090811 in PyModule_AddStringConstant (m=<module at remote 0x7fc7759b56a8>, name=0x7fc76af55ec5 "XML_ERROR_UNDECLARING_PREFIX", value=<value optimized out>) at Python/modsupport.c:636
        o = <value optimized out>
#3  0x00007fc76af5508d in initpyexpat () at /usr/src/debug/Python-2.6.6/Modules/pyexpat.c:2041
        m = <module at remote 0x7fc7759a6fd8>
        d = <value optimized out>
        errmod_name = <value optimized out>
        errors_module = <module at remote 0x7fc7759b56a8>
        modelmod_name = <value optimized out>
        model_module = <module at remote 0x7fc7759b5718>
        sys_modules = <value optimized out>
        capi = {magic = 0x0, size = 0, MAJOR_VERSION = 0, MINOR_VERSION = 0, MICRO_VERSION = 0, ErrorString = 0, GetErrorCode = 0, GetErrorColumnNumber = 0, GetErrorLineNumber = 0, Parse = 0, ParserCreate_MM = 0, ParserFree = 0, 
          SetCharacterDataHandler = 0, SetCommentHandler = 0, SetDefaultHandlerExpand = 0, SetElementHandler = 0, SetNamespaceDeclHandler = 0, SetProcessingInstructionHandler = 0, SetUnknownEncodingHandler = 0, SetUserData = 0}
        capi_object = <value optimized out>
#4  0x00007fc78308d865 in _PyImport_LoadDynamicModule (name=0x7fc77b631500 "pyexpat", pathname=0x7fc77b630430 "/usr/lib64/python2.6/lib-dynload/pyexpat.so", fp=<value optimized out>) at Python/importdl.c:53
        m = <value optimized out>
        lastdot = <value optimized out>
        shortname = <value optimized out>
        packagecontext = 0x0
        oldcontext = 0x0
        p = 0x7fc76af54ad0 <initpyexpat>
#5  0x00007fc78308b8a5 in import_submodule (mod=None, subname=0x7fc77b631500 "pyexpat", fullname=0x7fc77b631500 "pyexpat") at Python/import.c:2592
        buf = "/usr/lib64/python2.6/lib-dynload/pyexpat.so\000\000e.so\000-py2.6.egg/pyexpat.pyc\000e.so\000yc\000e.so\000y2.6.egg/pyexpat.pyc\000e.so\000.so\000\307\177\000\000om\376\202\307\177\000\000\020\005c{\307\177\000\000\020R1\203\307\177\000\000\001\000\000\000\000\000\000\000\251\262\r\203\307\177\000\000\251\262\r\203\307\177\000\000+x\376\202\307\177\000\000\220\026%\002\000\000\000\000\020R1\203\307\177\000\000\030\000\000\000\060\000\000\000\340\005c{\307\177\000\000$\000\000\000\307\177\000\000\t\000\000\000\000\000\000\000"...
        fp = 0x7fc775a970e0
        path = <value optimized out>
        loader = 0x0
        fdp = <value optimized out>
        modules = 
    {'reviewboard.diffviewer.django': None, 'django.utils.deprecation': <module at remote 0x7fc774dddb08>, 'reviewboard.diffviewer.parser': <module at remote 0x7fc775496520>, 'pygments.filters': <module at remote 0x7fc775b5c050>, 'haystack.signals': <module at remote 0x7fc774f69948>, 'django.dispatch.threading': None, 'docutils.utils.warnings': None, 'pprint': <module at remote 0x7fc774600408>, 'django.middleware.gzip': <module at remote 0x7fc774ba8ad0>, 'django.dispatch.weakref': None, 'paramiko.proxy': <module at remote 0x7fc7756aa520>, 'django.utils.log': <module at remote 0x7fc774b2a750>, 'paramiko.select': None, 'reviewboard.admin.haystack': None, 'SocketServer': <module at remote 0x7fc774b9dcc8>, 'django.contrib.admin.decimal': None, 'xml.etree.sys': None, 'django.test.contextlib': None, 'shlex': <module at remote 0x7fc775550d00>, 'django.utils.http': <module at remote 0x7fc774a31948>, 'djblets.util.datetime': None, 'django.conf.urls.django': None, 'json.sre_constants': None, 'django.core.serializers': <mod...(truncated)
        m = <value optimized out>
#6  0x00007fc78308bb24 in load_next (mod=None, altmod=None, p_name=<value optimized out>, buf=0x7fc77b631500 "pyexpat", p_buflen=0x7fc77b6314f8) at Python/import.c:2412
        name = 0x7fc775a52b64 "pyexpat"
        dot = <value optimized out>
        len = 7
        p = 0x7fc77b631500 "pyexpat"
        result = <value optimized out>
#7  0x00007fc78308c17d in import_module_level (name=0x0, globals=<value optimized out>, locals=<value optimized out>, fromlist=['__doc__'], level=<value optimized out>) at Python/import.c:2133
        buf = "pyexpat\000\004\000\000\000\000\000\000\000\340\315\301u\307\177\000\000\260\367\277u\307\177\000\000\017\000\000\000\000\000\000\000p\352\230u\307\177\000\000\260\001{\207\307\177\000\000\345,\001\203\307\177\000\000p#c{\307\177\000\000t\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\250\351\b\203\307\177\000\000ponent(\001\000\000\000t\t\000\000\000universali\002\000\000\000i\005\000\000\000(\001\000\000\000t\n\000\000\000__import__t\006\000\000\000Writerc\000\000\000\000\000\000\000\000\001\000\000\000B\000\000\000s_\000\000\000e\000\000Z\001\000d\000\000Z\002\000d\001\000Z\003\000d\002\000Z\004\000d\003\000\204\000\000Z\005\000d\b\000Z\a\000"...

It seems to be related to pyexpat, and I did read about those issues here: https://code.google.com/p/modwsgi/wiki/IssuesWithExpatLibrary . However, I'm running Python 2.6.6, and it says there that those issues only apply to versions of Python prior to 2.5.

Answer 1

You've already diagnosed the answer yourself. It is the same problem I was having.

• Using Oracle Enterprise Linux 6 which has /lib64/libexpat.so.1.5.2
• Using Oracle HTTP Server 12c (essentially a rebranded Apache), which has <install path>/ohs/lib/libexpat.so.0.5.0

Like you, I did a backtrace of the Apache httpd coredump. Apache crashed in __strlen_ss43 while calling initpyexpat , as part of loading the Python module pyexpat , which was being loaded because ReviewBoard needed it.

As you linked, it's a known issue caused by the Python pyexpat module being compiled for one version of libexpat , but an incompatible version is loaded .

On my system, the pyexpat module (installed as a dependency of ReviewBoard) is compiled and linked against the system's /lib64/libexpat.so.1.5.2 . However, Apache is configured to load libexpat.so.0.5.0 .

The linked article say that the problem is fixed in Python 2.6, but not the Python 2.6 in OEL 6:

$ ldd /usr/lib64/python2.6/lib-dynload/pyexpat.so
    linux-vdso.so.1 =>  (0x00007fff0c7de000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f0f27dc0000)
    ...

The pyexpat module here isn't name prefixed as the linked article says, it's just using the system-wide libexpat , so anything will do.

I found that I do not use anything in the Apache setup which requires libexpat, except for ReviewBoard. So I am justified in removing/replacing the libexpat.so links in the Apache install directory with the one that pyexpat was compiled for:

# cd <install path>/ohs/lib
# mkdir disabled
# mv libexpat* disabled
# ln -s /lib64/libexpat.so.1 libexpat.so.0