Using Pyramid events and multithreading

Question

I'd like to use events subscription / notification together with multithreading. It sounds like it should just work in theory and the documentation doesn't include any warnings. The events should be synchronous, so no deferring either.

But in practice, when I notify off the main thread, nothing comes in:

def run():
    logging.config.fileConfig(sys.argv[1])
    with bootstrap(sys.argv[1]) as env:
        get_current_registry().notify(FooEvent())  # <- works
        Thread(target=thread).start()              # <- doesn't work

def thread():
    get_current_registry().notify(FooEvent())

Is this not expected to work? Or am I doing something wrong?

I tried also the suggested solution. It doesn't print the expected event.

class Foo:
    pass

@subscriber(Foo)
def metric_report(event):
    print(event)

def run():
    with bootstrap(sys.argv[1]) as env:

        def foo(env):
            try:
                with env:
                    get_current_registry().notify(Foo())
            except Exception as e:
                print(e)

        t = Thread(target=foo, args=(env,))
        t.start()
        t.join()

Answer 1

get_current_registry() is trying to access the threadlocal variable Pyramid registers when processing requests or config to tell the thread what Pyramid app is currently active IN THAT THREAD. The gotcha here is that get_current_registry() always returns a registry, just not the one you want, so it's hard to see why it's not working.

When spawning a new thread, you need to register your Pyramid app as the current threadlocal. The best way to do this is with pyramid.scripting.prepare . The "easy" way is just to run bootstrap again in your thread. I'll show the "right" way though.

def run():
    pyramid.paster.setup_logging(sys.argv[1])
    get_current_registry().notify(FooEvent())  # doesn't work, just like in the thread
    with pyramid.paster.bootstrap(sys.argv[1]) as env:
        registry = env['registry']
        registry.notify(FooEvent())  # works
        get_current_registry().notify(FooEvent())  # works
        Thread(target=thread_main, args=(env['registry'],)).start()

def thread_main(registry):
    registry.notify(FooEvent())  # works, but threadlocals are not setup if other code triggered by this invokes get_current_request() or get_current_registry()

    # so let's setup threadlocals
    with pyramid.scripting.prepare(registry=registry) as env:
        registry.notify(FooEvent())  # works
        get_current_registry().notify(FooEvent())  # works

pyramid.scripting.prepare is what bootstrap uses under the hood, and is a lot more efficient than running bootstrap multiple times because it shares the registry and all of your app configuration instead of making a completely new copy of your app.

Answer 2

Is it just that the 'with' context applies to the Thread() create statement only and does not propogate to the thread() method. ie in the case that works the 'get_current_registry' call has 'with' env context, but this 'with' context will not propogate to the point where the thread runs the 'get_current_registry'. So you need to propogate the env to the thread() - perhaps by creating a simple runnable class that takes the env in the init method.

class X:
    def __init__(self,env):
        self.env = env

    def __call__(self):
        with self.env:
            get_current_registry().notify(FooEvent())
        return

def run():
    logging.config.fileConfig(sys.argv[1])
    with bootstrap(sys.argv[1]) as env:
        get_current_registry().notify(FooEvent())
        Thread(target=X(env)).start()