mod_wsgi not finding modules - apache, RHEL, python3.4

Question

I had mod_wsgi working fine on RHEL/Apache2.4/python3.4/Django1.11/virtualenv. Then, I used pip within the virtualenv to upgrade numpy from 1.14.0 to 1.14.2, install pandas, and install a couple other modules. Then I restarted Apache and started getting server errors indicating AttributeError: 'module' object has no attribute 'arange' when trying to call numpy.arange , or similar errors when trying to call matplotlib.use('Agg'), etc.

I replaced wsgi.py with the following and it works fine. As soon as I uncomment any of the numpy or matplotlib calls it returns a 500 error and in the apache error log it shows an error similar to AttributeError: 'module' object has no attribute ...

import datetime
import sys
import numpy as np
import matplotlib as mpl
import mod_wsgi

def application(env, start_response):

    #mpl.use('Agg')
    #a = np.arange(15).reshape(3, 5)
    #a = np.array([2,3,4])

    status = '200 OK'
    output = 'Hello World! This is the wsgi app generated from python!'
    output += '\n\ncurrent time is: '+str(datetime.datetime.now())
    output += '\n\nsys.executable='+sys.executable
    output += '\n\nsys.path='+str(sys.path)
    output += '\n\nsys.version=' + str(sys.version)
    output += '\n\nsys.prefix=' +  str(sys.prefix)
    output += '\n\nmod_wsgi.version='+str(mod_wsgi.version)
    output += '\n\n\nEnvironment Variables:\n\n'+'\n'.join('%s: %s' % (k, v) for (k, v) in env.items())
    output = bytes(output,'utf-8')
    response_headers = [('Content-type', 'text/plain'),
                            ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]

Relevant output from this wsgi.py:

sys.executable=/usr/bin/python3
sys.path=['/usr/local/virtualenvs/myapp/wsgiapp', '/usr/local/virtualenvs/myapp/lib64/python34.zip', '/usr/local/virtualenvs/myapp/lib64/python3.4', '/usr/local/virtualenvs/myapp/lib64/python3.4/plat-linux', '/usr/local/virtualenvs/myapp/lib64/python3.4/lib-dynload', '/usr/lib64/python3.4', '/usr/lib/python3.4', '/usr/local/virtualenvs/myapp/lib/python3.4/site-packages']
sys.version=3.4.8 (default, Mar 23 2018, 10:04:27) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
sys.prefix=/usr/local/virtualenvs/myapp
mod_wsgi.version=(4, 5, 24)
mod_wsgi.process_group: mywsgiapp
mod_wsgi.application_group: 

The Apache httpd.conf didn't change. Relevant lines:

LoadModule wsgi_module "/usr/local/virtualenvs/myapp/lib64/python3.4/site-packages/mod_wsgi/server/mod_wsgi-py34.cpython-34m.so"
WSGIDaemonProcess mywsgiapp python-home=/usr/local/virtualenvs/myapp python-path=/usr/local/virtualenvs/myapp/wsgiapp
WSGIProcessGroup mywsgiapp
WSGIApplicationGroup %{GLOBAL}
WSGIScriptAlias /myapp /usr/local/virtualenvs/myapp/wsgiapp/wsgi.py process-group=mywsgiapp application-group=%{GLOBAL}

I'm not sure what caused this to break. I know this usually happens when mod_wsgi is compiled for the wrong version of python. But this was working fine before and everything still appears to be configured correctly: mod_wsgi is functioning, the output from the wsgi.py indicates it is using the correct version of python, and the site-packages directory of the virtualenv is in the python-path.

I have mod_wsgi installed in the main python3.4 location and tried pointing the Apache LoadModule at that instance but still get the same errors.

What else should I check?

Should sys.executable be pointing to the virtualenv instead of the main python install? Should I pip uninstall all the modules and start over?

Also, to install pandas, I had to install Cython, which in turn required that I install gcc-c++ ( sudo yum install gcc-c++ ). Would that affect anything?

Edited to add error traces. I uncommented the line a = np.arange(15).reshape(3, 5) in the above wsgi.py file.

HTTP Error Page from Apache:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Apache error log (note I x'ed out the IP address before posting here):

[Thu Jul 12 23:01:15.664941 2018] [wsgi:error] [pid 21490] [remote 10.x.x.x:59961] mod_wsgi (pid=21490): Exception occurred processing WSGI script '/usr/local/virtualenvs/myapp/wsgiapp/wsgi.py'.
[Thu Jul 12 23:01:15.727450 2018] [wsgi:error] [pid 21490] [remote 10.x.x.x:59961] Traceback (most recent call last):
[Thu Jul 12 23:01:15.727814 2018] [wsgi:error] [pid 21490] [remote 10.x.x.x:59961]   File "/usr/local/virtualenvs/myapp/wsgiapp/wsgi.py", line 14, in application
[Thu Jul 12 23:01:15.727831 2018] [wsgi:error] [pid 21490] [remote 10.x.x.x:59961]     a = np.arange(15).reshape(3, 5)
[Thu Jul 12 23:01:15.727865 2018] [wsgi:error] [pid 21490] [remote 10.x.x.x:59961] AttributeError: 'module' object has no attribute 'arange'

Error when trying to show np. file :

[Thu Jul 12 23:25:59.196534 2018] [wsgi:error] [pid 22747] [remote 10.x.x.x:60353] mod_wsgi (pid=22747): Exception occurred processing WSGI script '/usr/local/virtualenvs/myapp/wsgiapp/wsgi.py'.
[Thu Jul 12 23:25:59.257214 2018] [wsgi:error] [pid 22747] [remote 10.x.x.x:60353] Traceback (most recent call last):
[Thu Jul 12 23:25:59.257500 2018] [wsgi:error] [pid 22747] [remote 10.x.x:60353]   File "/usr/local/virtualenvs/myapp/wsgiapp/wsgi.py", line 23, in application
[Thu Jul 12 23:25:59.257535 2018] [wsgi:error] [pid 22747] [remote 10.x.x.x:60353]     output += '\\n\\nnumpy file location='+str(np.__file__)
[Thu Jul 12 23:25:59.257569 2018] [wsgi:error] [pid 22747] [remote 10.x.x.x:60353] AttributeError: 'module' object has no attribute '__file__'

Also, just for reference, importing the modules works just fine if I activate the virtualenv and use the interpreter:

Python 3.4.8 (default, Mar 23 2018, 10:04:27)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__file__
'/usr/local/virtualenvs/myapp/lib/python3.4/site-packages/numpy/__init__.py'

Printing dir(np) from the wsgi.py file results in:

dir(np)=['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

So it doesn't error out, but it is not the full list of identifiers. Running dir(np) from the virtualenv interpreter results in:

['ALLOW_THREADS', 'AxisError', 'BUFSIZE', 'CLIP', 'ComplexWarning', 'DataSource', 'ERR_CALL', ..... clipped for brevity - there's a full screen of items in the list

Since it does list some identifiers, here are the results of printing each of them from within wsgi.py. It does seem to be picking up the package from the correct path.

np.__path__ =_NamespacePath(['/usr/local/virtualenvs/myapp/lib/python3.4/site-packages/numpy'])
np.doc=None
np.loader=<_frozen_importlib._NamespaceLoader object at 0x7f83b82e0c88>
np.package=numpy
np.name=numpy
np.Spec=ModuleSpec(name='numpy', loader=None, origin='namespace', submodule_search_locations=_NamespacePath(['/usr/local/virtualenvs/myapp/lib/python3.4/site-packages/numpy']))

Just to be sure wsgi/python is truly finding the numpy package, I changed the wsgi.py file to import a non-existent package import foo as np , and when I ran it again it errored out with ImportError: No module named 'foo' . So it does seem to be actually finding the site-packages... but just not fully loading them?

Answer 1

Fun fact: when pip installs a new module it sets the permissions such that the user is able to import and use the modules as expected but "other" permissions allow the modules to be seen but not execute. Thus, mod_wsgi running under Apache was able to import the modules but calling any methods resulted in an error.

Solution : double check permissions after installing or updating python modules! I set all directories to 755 and all files to 644 and that seemed to work. Or, as mentioned in the comment, ensure that your umask is set appropriately before running pip.