UnboundExecutionError from SQLAlchemy in Cherrypy project using Nginx with uWsgi

Question

I am currently using SQLAlchemy as ORM in my Cherrypy project. The error that I have does not occur while I am on my cherrypy enbedded webserver, but ONLY from my Nginx deployment.

On Debian Wheezy with Nginx 1.2.1, but it does not matter as I've already thied on Sqeeze with Nginx 1.2.8 and 1.4 - same result!

So, I deploy my project with Nginx & uWsgi. My wsgi_handler.py does not differ from my devserver starting script, but anyway I have no errors in dev mode.

testproj:
        - models.py
        - root.py
        - tools.py //this is a cherrypy tool that I found to bind sqlalchemy session to cherrypy requests
        - devserver.py

        //the production mode files
        - wsgi_handler.py
        - uwsgi.ini

Here is a minimal configuration of a web project satisfying my application requirements:

models.py:

# -*- coding: utf-8 -*-
import logging
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column
from sqlalchemy.types import String, Integer

logging.basicConfig()
logging.getLogger('sqlalchemy.engine').setLevel(logging.CRITICAL)

Base = declarative_base()


class User(Base):
    __tablename__ = 'user'
    id = Column(Integer, primary_key=True)
    firstname = Column(String)
    lastname = Column(String)

    def __init__(self, firstname, lastname):
        self.firstname = firstname
        self.lastname = lastname

tools.py:

# -*- coding: utf-8 -*-
import os
import cherrypy

from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker

from models import *


class SAEnginePlugin(cherrypy.process.plugins.SimplePlugin):
    def __init__(self, bus, dburi):
        """
        The plugin is registered to the CherryPy engine and therefore
        is part of the bus (the engine *is* a bus) registery.

        We use this plugin to create the SA engine. At the same time,
        when the plugin starts we create the tables into the database
        using the mapped class of the global metadata.

        Finally we create a new 'bind' channel that the SA tool
        will use to map a session to the SA engine at request time.
        """
        cherrypy.process.plugins.SimplePlugin.__init__(self, bus)
        self.sa_engine = None
        self.bus.subscribe("bind", self.bind)
        self.dburi = dburi

    def start(self):
        db_path = os.path.abspath(os.path.join(os.curdir, self.dburi))
        self.sa_engine = create_engine('sqlite:///%s' % db_path, echo=True)
        Base.metadata.create_all(self.sa_engine)

    def stop(self):
        if self.sa_engine:
            self.sa_engine.dispose()
            self.sa_engine = None

    def bind(self, session):
        session.configure(bind=self.sa_engine)


class SATool(cherrypy.Tool):
    def __init__(self):
        """
        The SA tool is responsible for associating a SA session
        to the SA engine and attaching it to the current request.
        Since we are running in a multithreaded application,
        we use the scoped_session that will create a session
        on a per thread basis so that you don't worry about
        concurrency on the session object itself.

        This tools binds a session to the engine each time
        a requests starts and commits/rollbacks whenever
        the request terminates.
        """
        cherrypy.Tool.__init__(self, 'on_start_resource', self.bind_session, priority=20)

        self.session = scoped_session(sessionmaker(autoflush=True, autocommit=False))

    def _setup(self):
        cherrypy.Tool._setup(self)
        cherrypy.request.hooks.attach('on_end_resource', self.commit_transaction, priority=80)

    def bind_session(self):
        cherrypy.engine.publish('bind', self.session)
        cherrypy.request.db = self.session

    def commit_transaction(self):
        cherrypy.request.db = None
        try:
            self.session.commit()
        except:
            self.session.rollback()  
            raise
        finally:
            self.session.remove()

root.py:

# -*- coding: utf-8 -*-
import cherrypy
from models import User

class Root(object):
    def index(self):
        """
        List users from the DB & add new if no users
        """
        user = cherrypy.request.db.query(User).filter_by(firstname='John').first()
        if user is None:
            user = User('John', 'Doe')
            cherrypy.request.db.add(user)
            return 'Created user: %s %s' % (user.firstname, user.lastname)
        return 'Found user: %s %s' % (user.firstname, user.lastname)
    index.exposed = True

Here is , where my devserver.py does not differ so much from my wsgi_handler.py ，其中我的devserver.py与我的wsgi_handler.py并没有太大区别

devserver.py:

# -*- coding: utf-8 -*-
import os, cherrypy
from tools import SAEnginePlugin, SATool
from root import Root

basedir = os.path.abspath(os.path.dirname(__file__))

config = {
            'DEFAULT':  {
                            basedir : basedir,
            },

            'global':   {
                            'server.socket_port' : 8000,
                            'server.socket_host' : "127.0.0.1",

                            'tools.encode.on' : True,
                            'tools.encode.encoding' : 'utf-8',

                            'request.show_tracebacks' : True,
            },

            '/':        {
                            'tools.db.on' : True,
            },
}


if __name__ == "__main__":
    SAEnginePlugin(cherrypy.engine, '/tmp/data.db3').subscribe()

    cherrypy.tools.db = SATool()

    cherrypy.quickstart(Root(), config=config)

wsgi_handler.py:

# -*- coding: utf-8 -*-
import os, cherrypy
from tools import SAEnginePlugin, SATool
from root import Root

basedir = os.path.abspath(os.path.dirname(__file__))

config = {
            'DEFAULT':  {
                            basedir : basedir,
            },

            'global':   {
                            'server.socket_port' : 80,
                            'server.socket_host' : "127.0.0.1",

                            'tools.encode.on' : True,
                            'tools.encode.encoding' : 'utf-8',

                            'request.show_tracebacks' : True,
            },

            '/':        {
                            'tools.db.on' : True,
            },
}


def application(environ, start_response):
    SAEnginePlugin(cherrypy.engine, '/tmp/data.db3').subscribe()

    cherrypy.tools.db = SATool()

    cherrypy.tree.mount(Root(), script_name='/', config=config)
    return cherrypy.tree(environ, start_response)

uwsgi.ini:

[uwsgi]
thread=3
master=1
uid = www-data
gid = www-data
wsgi-file = /var/www/testproj/wsgi_handler.py
chdir = /var/www/testproj
socket = /tmp/uwsgi.sock
logto = /var/log/uwsgi.log

sudo apt-get install python-dev python-setuptools nginx
sudo easy_install pip
sudo pip install CherryPy==3.2.2 SQLAlchemy==0.7.10 uWSGI==1.9.11

python devserver.py

localhost:8000 gives me the first & the last name of the registered user in my database. If it does not exist, I create it & display it.

sudo mkdir -p /var/www
sudo cp -r testproj /var/www/
sudo chown -R www-data:www-data /var/www/testproj

Create the nginx configuration file:

/etc/nginx/sites-available/default:

server {
    server_name 127.0.0.1;

    listen      80;

    charset UTF-8;
    access_log  /var/log/nginx/access.log;
    error_log   /var/log/nginx/error.log;

    location / {
        uwsgi_pass unix:///tmp/uwsgi.sock;
        include uwsgi_params;
    }
}

sudo service nginx restart
uwsgi --enable-threads --ini /var/www/testproj/uwsgi.ini

On localhost:8000 I get the expected output:

Found user: John Doe

And on

127.0.0.1/

I get:

I do not see why as in dev mode the session bounds as it should!

Answer 1

The wsgi_handler.py script is wrong. You are trying to reinitialize your whole cherrypy stack everytime, while you should create it on startup and then call the WSGI callable.

Basically you should move all of the code out of 'application' and change the last line as

application = cherrypy.tree

In addition to this SQLAlchemy is not fork() friendly so you should always completely reload your app in each worker. Adding --lazy-apps will do the trick

Answer 2

Your SAEnginePlugin.subscribe() never fires the start() method, as no cherrypy engine has been started.

The end of your wsgi_handler.py should look like:

SAEnginePlugin(cherrypy.engine, '/tmp/data.db3').subscribe()
cherrypy.tools.db = SATool()
cherrypy.tree.mount(Root(), script_name='/', config=config)

cherrypy.engine.start()

application = cherrypy.tree