简体   繁体   English

如何使用 Boto3 和 Python 将多个批处理消息从列表发送到 SQS 队列

[英]How to send multiple batch messages from a list to an SQS queue using Boto3 and Python

I have a list of integers (integerList) that I'd like to pass into an SQS queue where each message into the queue is an integer from the list.我有一个整数列表 (integerList),我想将其传递到 SQS 队列中,其中进入队列的每条消息都是列表中的 integer。

I can do this one message at a time with the send_message() command, and the code for that is below.我可以使用send_message()命令一次发送一条消息,代码如下。

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    for i in integerList:
        response = queue.send_message(MessageBody=str(i),
                                      MessageGroupId='TESTING')

However, I'd like to speed up the function and send the messages in batches.不过,我想加快function的速度,批量发送消息。 Currently, AWS SQS allows batching up to 10 messages at a time with the send_messages() command, but I'm not sure how to build the Entries= attribute for the batch send.目前,AWS SQS 允许使用send_messages()命令一次最多批处理 10 条消息,但我不确定如何为批量发送构建Entries=属性。 I'm breaking down the integerList into smaller lists of 10 using chunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)] , but the next steps are unclear.我使用chunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)]将 integerList 分解为更小的 10 列表,但接下来的步骤尚不清楚。

According to the documentation Entries is a list of messages. 根据文档, Entries是一个消息列表。
For each entry in Entries the parameters type are detailed at the link. 对于每个条目Entries的参数类型是在链路详细。

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    entries = []

    for i in integerList:
        entry =  {
            'Id': 'id%s' % str(integerList[i]),
            'MessageBody': str(integerList[i])
            }
        entries.append(entry)

    response = queue.send_messages(entries)

Building on the answer from @liorko and some trial and error, this seems to work and is much faster than the 1-by-1 method I was using before. 基于@liorko的答案和一些反复试验,这似乎有效,并且比我之前使用的1-by-1方法快得多。

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    maxBatchSize = 10 #current maximum allowed
    chunks = [integerList[x:x+maxBatchSize] for x in range(0, len(integerList), maxBatchSize)]
    for chunk in chunks:
        entries = []
        for x in chunk:
            entry = {'Id': str(x), 
                     'MessageBody': str(x), 
                     'MessageGroupId': 'ANYTHINGYOUWANT'}
            entries.append(entry)
        response = queue.send_messages(Entries=entries)

According to the new version of boto3, the method to use is send_message_batch , here is the updated code:根据boto3的新版本,使用的方法是send_message_batch ,这里是更新的代码:

import boto3

sqs_client = boto3.client("sqs", region_name="eu-west-1")

entries = []
for i in integer_list:
    entry =  {
        'Id': 'id%s' % str(integerList[i]),
        'MessageBody': str(integerList[i])
        }
    entries.append(entry)
    response = queue.send_message_batch(Entries=entries, QueueUrl="https://sqs.eu-west-1.amazonaws.com/000000000000/my-queue-name")

Source: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.send_message_batch资料来源: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.send_message_batch

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM