RabbitMQ Python Pika - 多条消息的连接处理

Question

I have been reading multiple blogs and documentation about proper setting up RabbitMQ connection for publishing. Below is my scenario

• Have few scheduled jobs which execute certain tasks and publish the output to RabbitMQ
• The jobs runs at different time intervals, but the output will be posted to same RabbitMQ queue

The following is the implementation

def get_credentials(self):
    print ("Host ", config_reader.get_lookup_data('RABBITMQ', 'host'))
    credentials = pika.PlainCredentials(config_reader.get_lookup_data('RABBITMQ', 'user'),
                                        config_reader.get_lookup_data('RABBITMQ', 'password'))
    return credentials

def publish_message(self, message):
    print ("Publish message" , message)
    connection = pika.BlockingConnection(
        pika.ConnectionParameters(host=config_reader.get_lookup_data('RABBITMQ', 'host'),
                                  credentials=self.get_credentials()))
    channel = connection.channel()
    channel.exchange_declare(exchange=config_reader.get_lookup_data('RABBITMQ', 'exchange'), passive=True)
    result = channel.queue_declare(exclusive=False,
                                   queue=config_reader.get_lookup_data('RABBITMQ', 'sensor_queue'))
    channel.queue_bind(result.method.queue,
                       exchange=config_reader.get_lookup_data('RABBITMQ', 'exchange'),
                       routing_key=config_reader.get_lookup_data('RABBITMQ', 'routing_key'))
    print ('Publishing message ', message)
    channel.basic_publish(exchange=config_reader.get_lookup_data('RABBITMQ', 'exchange'), body=json.dumps(message),
                          routing_key=config_reader.get_lookup_data('RABBITMQ', 'routing_key'),
                          properties=pika.BasicProperties(
                              headers={'Content-Type': 'application/json'}  # Add a key/value header
                          ))
    print ('published')

I observe that the above implementation is every job is establishing a connection and then a channel. I doubt if this type of implementation is causing unnecessary overhead.

Can someone suggest right way to handle the connection object. I personally feel creating connection for every message is certainly overhead

Answer 1

https://www.rabbitmq.com/tutorials/amqp-concepts.html#amqp-connections

TL;DR

AMQP connections are long-lived. The handshake process for an AMQP connection is quite complex and requires at least 7 TCP packets (more if TLS is used). A best practice is to reuse connections and multiplex a connection between threads with channels.

Connection pool details: A connection pool with a minimum of 10 connections. If more than 10 are needed a new connection can be created. The maximum connections can be of 40 in the pool. A time limit can be set after which the connection can be closed so that the connections can be closed instead of being there forever.

Reference: https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-common-errors.html