如何从python中的线程池中唤醒线程？

Question

I new to Python and am developing an application in Python 2.7. I am using a thread pool provided by the concurrent.futures library. Once a thread from ThreadPool is started, it needs to wait for some message from RabbitMQ.

How can I implement this logic in Python to make this thread from the pool wait for event messages? Basically I need to wake up a waiting thread once I receive message from RabbitMQ (ie wait and notify implementation on ThreadPool ).

Answer 1

First you define a Queue :

from Queue import Queue

q = Queue()

then, in your thread, you attempt to get an item from that queue:

msg = q.get()

this will block the entire thread until there is something to be found in the queue.

Now, at the same time, assuming your incoming events are notified by means of triggering callbacks, you register a callback that simply puts the received RabbitMQ message in the queue:

def on_message(msg):
    q.put(msg)
rabbitmq_channel.register_callback(on_message)

or if you like shorter code:

rabbitmq_channel.register_callback(lambda msg: q.put(msg))

(the above is pseudocode because I've not used RabbitMQ nor whatever Python bindings for RabbitMQ, but you should be able to easily figure out how to adapt the snippet to your real application code; the key part to pay attention to is q.put(msg) —just make sure that part gets invoked as soon as a new message is notified.)

as soon as this happens, the thread is awakened and is free to process the message. In order to reuse the same thread for multiple messages, just use a while loop:

while True:
    msg = q.get()
    process_message(msg)

PS I would suggest looking into Gevent and how to combine it with RabbitMQ in your Python application so as to be able to get rid of threads and use more lightweight and scalable green threading mechanism instead without ever having to manage a threadpool (because you can just have tens of thousands of greenlets spawned and killed on the fly):

# this thing always called in a green thread; forget about pools and queues.
def on_message(msg):
    # you're in a green thread now; just process away!
    benefit_from("all the gevent goodness!")
    spawn_and_join_10_sub_greenlets()

rabbitmq_channel.register_callback(lambda msg: gevent.spawn(on_message, msg))