繁体   English   中英

使用无限循环不断轮询 SQS 队列

[英]Constantly polling SQS Queue using infinite loop

我有一个 SQS 队列,我需要不断监控传入消息。 一旦消息到达,我会进行一些处理并继续等待下一条消息。 我通过在循环结束时设置一个 2 秒暂停的无限循环来实现这一点。 这可行,但是我不禁觉得这不是解决不断排队的需要的一种非常有效的方法。

代码示例:

    while (1):
        response = sqs.receive_message(
            QueueUrl=queue_url,
            AttributeNames=[
                'SentTimestamp'
            ],
            MaxNumberOfMessages=1,
            MessageAttributeNames=[
                'All'
            ],
            VisibilityTimeout=1,
            WaitTimeSeconds=1
        )

        try:
            message = response['Messages'][0]
            receipt_handle = message['ReceiptHandle']

            # Delete received message from queue
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=receipt_handle
            )
            msg = message['Body']
            msg_json = eval(msg)
            value1 = msg_json['value1']
            value2 = msg_json['value2']
            process(value1, value2)
        except:
            pass
            #print('Queue empty')
        time.sleep(2)

为了干净地退出脚本(应该不断运行),我捕获了在 Ctrl+C 上触发的 KeyboardInterrupt 并执行一些清理例程以优雅地退出。

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        logout()

有没有更好的方法来实现 SQS 队列的恒定轮询,是否需要 2 秒的延迟? 我试图不锤击 SQS 服务,但也许没关系?

这最终是 SQS 的工作方式——它需要一些东西来轮询它以获取消息。 但一些建议:

不要每次只收到一条消息。 做更多类似的事情:

messages = sqs.receive_messages(
        MessageAttributeNames=['All'],
        MaxNumberOfMessages=10,
        WaitTimeSeconds=10
    )
for msg in messages:
    logger.info("Received message: %s: %s", msg.message_id, msg.body)

这对你来说有点改变。 第一件事是您愿意接收最多 10 条消息(这是 SQS 在一次调用中的最大数量)。 第二个是您将等待最多 10 秒来获取消息。 来自 SQS 文档:

调用在返回之前等待消息到达队列的持续时间(以秒为单位)。 如果消息可用,则调用会比 WaitTimeSeconds 更快返回。 如果没有消息可用并且等待时间到期,则调用成功返回,并带有一个空的消息列表。

因此,您不需要自己的sleep呼叫 - 如果没有消息,呼叫将等到它过期。 相反,如果您有大量消息,那么您将尽可能快地获取它们,因为您不会在代码中拥有自己的sleep调用。

添加@stdunbar 答案:

您会发现文档中所述的MaxNumberOfMessages可能返回的消息少于提供的整数,这对我来说就是这种情况。

MaxNumberOfMessages (integer) -- 要返回的最大消息数。 Amazon SQS 返回的消息永远不会超过此值(但是,返回的消息可能会更少)。 有效值:1 到 10。默认值:1。

结果,我做了这个从 SQS Dead-Letter-Queue 读取的解决方案:

def read_dead_letter_queue():

""" This function is responsible for Reading Query Execution IDs related to the insertion that happens on Athena Query Engine
         and we weren't able to deal with it in the Source Queue.

Args: 
    None

Returns:
    Dictionary: That consists of execution_ids_list, mssg_receipt_handle_list and queue_url related to messages in a Dead-Letter-Queue that's related to the insertion operation into Athena Query Engine.
"""

  try:
    sqs_client = boto3.client('sqs')
    
    queue_url = os.environ['DEAD_LETTER_QUEUE_URL'] 
    
    execution_ids_list = list()
    
    mssg_receipt_handle_list = list()
    
    final_dict = {}
    
    # You can change the range stop number to whatever number that suits your scenario, you just need to add a number that's more than the number of messages that maybe in the Queue as 1 thousand or 1 million, as the loop will break out when there aren't any messages left in the Queue before reaching the end of the range.
    for mssg_counter in range(1, 20, 1):
        
        sqs_response = sqs_client.receive_message(
            QueueUrl = queue_url,
            MaxNumberOfMessages = 10,
            WaitTimeSeconds = 10
        )
        
        print(f"This is the dead-letter-queue response --> {sqs_response}")
        
        try:
            for mssg in sqs_response['Messages']:
                    print(f"This is the message body --> {mssg['Body']}")
                    print(f"This is the message ID --> {mssg['MessageId']}")
                    execution_ids_list.append(mssg['Body'])
                    mssg_receipt_handle_list.append(mssg['ReceiptHandle'])
        except:
            print(f"Breaking out of the loop, as there isn't any message left in the Queue.")
            break
    
    print(f"This is the execution_ids_list contents --> {execution_ids_list}")
    
    print(f"This is the mssg_receipt_handle_list contents --> {mssg_receipt_handle_list}")
    
    # We return the ReceiptHandle to be able to delete the message after we read it in another function that's responsible for deletion.
    # We return a dictionary consists of --> {execution_ids_list: ['query_exec_id'], mssg_receipt_handle_list: ['ReceiptHandle']}
    
    final_dict['execution_ids_list'] = execution_ids_list
    final_dict['mssg_receipt_handle_list'] = mssg_receipt_handle_list
    final_dict['queue_url'] = queue_url
    
    return final_dict
    
    #TODO: We need to delete the message after we finish reading in another function that will delete messages for both the DLQ and the Source Queue.
        
    
  except Exception as ex:
    print(f"read_dead_letter_queue Function Exception: {ex}")

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM