[英]RabbitMQ Python Pika - Connection handling for multiple messages
[英]RabbitMq - pika - python - Dropping messages when published
def get_connection_and_channel(self, connection_parameters):
connection = pika.BlockingConnection(connection_parameters)
channel = connection.channel()
return (connection, channel)
connection_parameters = pika.ConnectionParameters( server, port, virtual_host, credentials=pika.PlainCredentials(user_name, password))
connection,channel = self.get_connection_and_channel(connection_parameters)
channel.confirm_delivery()
count=0
for json_string in open(json_file, 'r'):
result_json = json.loads(json_string)
message_body = json.dumps(result_json['body'])
routing_key = result_json['RoutingKey']
channel.basic_publish(exchange=self.output_exchange_name,routing_key=routing_key,body=message_body.strip())
count += 1
self.logger.info('Sent %d messages' % count)
connection.close()
我正在使用此代码将消息发送到RabbitMQ服务器。 但偶尔这不会将所有消息发送到相应的队列。 它每次运行时都会丢失随机数量的消息。
我无法理解这里的问题是什么。
有可能是您的消息被返回,因为它无法将消息路由到任何现有队列。 尝试在channel.confirm_delivery
添加回调:
channel.confirm_delivery(on_delivery_confirmation)
def on_delivery_confirmation(self, method_frame):
confirmation_type = method_frame.method.NAME.split('.')[1].lower()
if confirmation_type == 'ack':
self.logger.info('message published')
elif confirmation_type == 'nack':
self.logger.info('message not routed')
如果是这种情况,则在发布消息之前尝试首先使用交换和路由密钥绑定使用者队列。
首先,启用持久队列添加:
channel.queue_declare(queue='your_queue', durable=True)
发布者和消费者 (在进行发布/消费之前) 。
然后,即使RabbitMQ服务器死机并重新启动,您也可以确保您的队列不会丢失。
在发布者上,将properties=pika.BasicProperties(delivery_mode=2)
到您的basic_publish
调用中,以确保您的消息是持久的。
channel.basic_publish(exchange=self.output_exchange_name,
routing_key=routing_key,
body=message_body.strip(),
properties=pika.BasicProperties(delivery_mode=2))
这应该可以避免丢失_published消息。
从消费者的角度来看, python的官方RabbitMQ教程说:
为了确保消息永不丢失,RabbitMQ支持消息确认。 从消费者发回ack(nowledgement)以告知RabbitMQ已收到,处理了特定消息,并且RabbitMQ可以自由删除它。 [...]默认情况下会启用消息确认。
构建使用者时,确保正确发送ack ,让RabbitMQ将其从队列中删除。
def callback(ch, method, properties, body):
print "Received %r" % (body,)
ch.basic_ack(delivery_tag = method.delivery_tag)
channel.basic_consume(callback, queue='your_queue')
如果您需要一种更强大,更可靠的方法来完全确定在RabbitMQ上发布确认中继,您应该使用AMQP协议的plublish confirm功能。
来自pika文件 :
import pika
# Open a connection to RabbitMQ on localhost using all default parameters
connection = pika.BlockingConnection()
# Open the channel
channel = connection.channel()
# Declare the queue
channel.queue_declare(queue="test", durable=True, exclusive=False, auto_delete=False)
# Turn on delivery confirmations
channel.confirm_delivery()
# Send a message
if channel.basic_publish(exchange='test',
routing_key='test',
body='Hello World!',
properties=pika.BasicProperties(content_type='text/plain',
delivery_mode=1)):
print 'Message publish was confirmed'
else:
print 'Message could not be confirmed'
所以根据你的代码,我将使用类似的东西:
count=0
for json_string in open(json_file, 'r'):
result_json = json.loads(json_string)
message_body = json.dumps(result_json['body'])
routing_key = result_json['RoutingKey']
if channel.basic_publish(exchange=self.output_exchange_name,routing_key=routing_key,body=message_body.strip(),
properties=pika.BasicProperties(delivery_mode=2)): # Make it persistent
count += 1
else:
# Do something with your undelivered message
self.logger.info('Sent %d messages' % count)
connection.close()
或者作为一种强力方法,您可以使用while
循环而不是if
来确保发送所有消息:
count = 0
for json_string in open(json_file, 'r'):
result_json = json.loads(json_string)
message_body = json.dumps(result_json['body'])
routing_key = result_json['RoutingKey']
while not channel.basic_publish(exchange=self.output_exchange_name,
routing_key=routing_key,
body=message_body.strip(),
properties=pika.BasicProperties(delivery_mode=2)):
pass # Do nothing or even you can count retries
count += 1
self.logger.info('Sent %d messages' % count)
尝试使用您的命令只接收一条消息:
#!/usr/bin/env python
import pika
import ujson as json
def receive():
parameters = pika.ConnectionParameters(host='localhost')
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.queue_declare(queue='raw_post', durable=True)
method_frame, header_frame, body = channel.basic_get(queue='raw_post')
if method_frame.NAME == 'Basic.GetEmpty':
connection.close()
return ''
else:
channel.basic_ack(delivery_tag=method_frame.delivery_tag)
connection.close()
return json.loads(body), method_frame.message_count
a = ''
while a not in ['quit', 'sair', 'exit', 'bye']:
a = input("whats up?")
print(receive())
只有一个发送者有5000条消息要排队:
#!/usr/bin/env python
import pika
import ujson as json
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='raw_post', durable=True)
for i in range(5000):
info = {"info": "test", "value": i}
channel.basic_publish(exchange='',
routing_key='raw_post',
body=json.dumps(info),
properties=pika.BasicProperties(
delivery_mode=2, # make message persistent
))
print(" [x] Sent 'Hello World!' {}".format(i))
connection.close()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.