用c和mod_wsgi用apache扩展python / django

Question

I have a question about extending python with c code and mod_wsgi.

I have a django app in an apache server that query a postgresql database to generate reports. In some reports the system creates a csv file with the results. To create this csv file, some times the system must process more than two hundred thousands registers with python, that is obviously very slow. To accelerate this, we programed ac module that do this work, and that improves the speed many times. We have tried with ctypes and create a python module with c, both works fine in the runserver but crashes when executed with apache and mod_wsgi.

The error in httpd-error.log is:

[Wed Jul 27 02:33:52 2011] [notice] child pid 44657 exit signal Segmentation fault (11)

¿Any suggestion?

The code is:

# Creates the HttpResponse object with the appropriate CSV header.
response = HttpResponse(mimetype='application/x-zip-compressed')
response['Content-Disposition'] = \
    'attachment; filename=' + filename + '.zip'

p0 = 'descarga_' + str(datetime.today()) + '.csv'

p1 = settings.DATABASES['default']['NAME']
p2 = settings.DATABASES['default']['USER']

#lib.generar(string_at(p0),p1,p2,string_at(str(init)),string_at(str(end)),string_at(str(provider)))

import generador
generador.generar(p0,p1,p2,str(init),str(end),str(provider))

It crashes after we call generador.generar() that is an external module developed in C

I have also tried with GDb as @GrahamDumpleton suggets and this is the output, not very useful :(

The debugger starts and when i clic the link that executes that calls the c module it falls agin in segmentation fault

(gdb) run -X
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/local/sbin/httpd -X
[New LWP 101064]
[New Thread 28501140 (LWP 101064)]


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 28501140 (LWP 101064)]
0x2847d423 in fwrite () from /lib/libc.so.7

Well after the Segmetation fault, I tried with the 'where' command in the debugger and this is what it give:

(gdb) where
#0  0x2847d423 in fwrite () from /lib/libc.so.7
#1  0x293f8d09 in generar () from /ruta/al/codigo/generador.so
#2  0x28912caa in PyCFunction_Call () from /usr/local/lib/libpython2.7.so
#3  0x2896e49a in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.7.so
#4  0x2897044b in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.7.so
#5  0x288feafd in PyClassMethod_New () from /usr/local/lib/libpython2.7.so
#6  0x288d622c in PyObject_Call () from /usr/local/lib/libpython2.7.so
#7  0x2896c27a in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.7.so
#8  0x2896fadc in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.7.so
#9  0x2897044b in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.7.so
#10 0x288fea0a in PyClassMethod_New () from /usr/local/lib/libpython2.7.so
#11 0x288d622c in PyObject_Call () from /usr/local/lib/libpython2.7.so
#12 0x288e4fd8 in PyClass_IsSubclass () from /usr/local/lib/libpython2.7.so
#13 0x288d622c in PyObject_Call () from /usr/local/lib/libpython2.7.so
#14 0x2893044c in _PyObject_LookupSpecial () from /usr/local/lib/libpython2.7.so
#15 0x288d622c in PyObject_Call () from /usr/local/lib/libpython2.7.so
#16 0x28968ec4 in PyEval_CallObjectWithKeywords () from /usr/local/lib/libpython2.7.so
#17 0x2889b229 in Adapter_run (self=0x28b4dd58, object=0x28c7d50c) at mod_wsgi.c:3841
#18 0x2889be50 in wsgi_execute_script (r=0x28cb4058) at mod_wsgi.c:6547
#19 0x2889de66 in wsgi_hook_handler (r=0x28cb4058) at mod_wsgi.c:9080
#20 0x08076b19 in ap_run_handler (r=0x28cb4058) at config.c:157
#21 0x08079dee in ap_invoke_handler (r=0x28cb4058) at config.c:376
#22 0x08084eb0 in ap_process_request (r=0x28cb4058) at http_request.c:282
#23 0x0808201b in ap_process_http_connection (c=0x28b201f0) at http_core.c:190
#24 0x0807de09 in ap_run_process_connection (c=0x28b201f0) at connection.c:43
#25 0x08089791 in child_main (child_num_arg=Variable "child_num_arg" is not available.
) at prefork.c:662
#26 0x080899e3 in make_child (s=0x28510f10, slot=0) at prefork.c:707
#27 0x0808a591 in ap_mpm_run (_pconf=0x2850f018, plog=0x2853d018, s=0x28510f10) at  prefork.c:983
#28 0x08064195 in main (argc=676384792, argv=0x28b1e018) at main.c:739

Answer 1

Try setting:

WSGIApplicationGroup %{GLOBAL}

and forcing the application to run in main Python interpreter.

Possible your C extension isn't written properly to work with Python sub interpreters.

See:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

Your code could also just be buggy and using it under mod_wsgi shows out the problem where command line Python does.

See:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB

on how to debug where the crash is occurring.

---

UPDATE 1

When building your extension module, to aid debugging in gdb, ensure that optimisation isn't turned on when compiling extension module and that debug options are.

For setup.py driven extension module build, I find it necessary to add:

from distutils import sysconfig
dummy = sysconfig.get_config_vars('CFLAGS', 'OPT')
config_vars = sysconfig._config_vars
config_vars['CFLAGS'] = config_vars['CFLAGS'].replace(' -Os ', ' ')
config_vars['OPT'] = config_vars['OPT'].replace(' -Os ', ' ')

That way can get rid of -Os options. Whether it is -Os or -O will depend on your Python installation.

Check then for -g in compiler options when extension module built and add it if need be.

Once that is done you can then use gdb to actually dump out values of variables, get actual line numbers etc.