Pyhon Pika how to use GeventConnection

Question

We have several tasks that we consume from a message queue. The runtimes of those tasks are dependent on fetching some data from a database. Therefore we would like to work with Gevent to not block the program if some database requests take a long time. We are trying to couple it with the Pika client, which has some asynchronous adapters, one of them for gevent: pika.adapters.gevent_connection.GeventConnection .

I set up some toy code, which consumes from a MQ tasks that consists of integers and publishes them on another queue, while sleeping for 4 seconds for each odd number:

# from gevent import monkey
# # Monkeypatch core python libraries to support asynchronous operations.
# monkey.patch_time()

import pika
from pika.adapters.gevent_connection import GeventConnection
from datetime import datetime
import time


def handle_delivery(unused_channel, method, header, body):
    """Called when we receive a message from RabbitMQ"""
    print(f"Received: {body} at {datetime.now()}")
    channel.basic_ack(method.delivery_tag)

    num = int(body)
    print(num)

    if num % 2 != 0:
        time.sleep(4)

    channel.basic_publish(
        exchange='my_test_exchange2',
        routing_key='my_test_queue2',
        body=body
    )

    print("Finished processing")


def on_connected(connection):
    """Called when we are fully connected to RabbitMQ"""
    # Open a channel
    connection.channel(on_open_callback=on_channel_open)


def on_channel_open(new_channel):
    """Called when our channel has opened"""
    global channel
    channel = new_channel
    channel.basic_qos(prefetch_count=1)
    channel.queue_declare(queue="my_queue_gevent5")
    channel.exchange_declare("my_test_exchange2")
    channel.queue_declare(queue="my_test_queue2")
    channel.queue_bind(exchange="my_test_exchange2", queue="my_test_queue2")
    channel.basic_consume("my_queue_gevent5", handle_delivery)


def start_loop(i):
    conn = GeventConnection(pika.ConnectionParameters('localhost'), on_open_callback=on_connected)
    conn.ioloop.start()


start_loop(1)

If I run it without the monkey.patch_time() call it works OK and it publishes results on the my_test_queue2 , but it works sequentially. The expected behaviour after adding monkey.patch_time() patch would be that it still works but concurrently. However, the code gets stuck (nothing happens anymore) after it comes to the call time.sleep(4) . It processes and publishes the first integer, which is 0, and then gets stuck at 1, when the if clause gets triggered. What am I doing wrong?

Answer 1

With the help of ChatGPT I managed to make it work. There was a gevent.spawn() call missing:

def handle_delivery(unused_channel, method, header, body):
    print("Handling delivery")
    gevent.spawn(process_message, method, body)


def process_message(method, body):
    print(f"Received: {body} at {datetime.now()}")
    channel.basic_ack(method.delivery_tag)

    num = int(body)
    print(num)

    if num % 2 != 0:
        time.sleep(4)

    channel.basic_publish(
        exchange='my_test_exchange2',
        routing_key='my_test_queue2',
        body=body
    )

    print("Finished processing")