[英]AWS Lambda multiple threads and boto3
I am writing a AWS Lambda function that deletes 100,000 objects per lambda function call from S3 bucket. 我正在编写一个AWS Lambda函数,该函数从S3存储桶中每个lambda函数调用中删除100,000个对象。 I am trying to see if I can create and run the deletion on a background threads.
我正在尝试查看是否可以在后台线程上创建并运行删除操作。 I have the following code.
我有以下代码。
import boto3
import boto3.session
from threading import Thread
http_response = []
MAX = 999
threads = []
class myThread(Thread):
def __init__(self, objects_to_delete, bucket_name):
Thread.__init__(self)
self.objects_to_delete = objects_to_delete
self.bucket_name = bucket_name
def run(self):
session = boto3.session.Session().client('s3')
s3 = session.client('s3')
####
COMES HERE AND PRINTS THE NAME OF THE BUCKET.
####
print(self.bucket_name)
response = s3.delete_objects(Bucket=bucket_name, Delete={'Objects': objects_to_delete[0:MAX] })
####
THIS IS NOT GETTING PRINTED. MEANING, delete_object IS BREAKING/NOT EXECUTING
####
print(response)
def handler(event, context):
keys = event['keys']
bucket_name = event["bucket"]
if (len(keys) == 0 or len(bucket_name) == 0):
return {
"message": http_response
}
try:
t = myThread(objects_to_delete[0:MAX], bucket_name)
t.start()
threads.append(t)
except:
print("Something Went wrong!!! " + str(objects_to_delete))
del keys[0:MAX]
for i in range(len(threads)):
threads[i].start()
handler({'keys': keys, 'bucket': bucket_name}, context)
Is there anything wrong I am doing here? 我在这里做错什么吗? Seems like thread is starting, however it's not making the "delete_objects" call.
似乎线程正在启动,但是没有进行“ delete_objects”调用。 It's not even returning any error messages to learn about the error.
它甚至不返回任何错误消息以了解该错误。 Any thoughts or ideas?
有什么想法或想法吗?
One more thing, when I run this function locally on my computer, it runs just fine without any problem. 还有一件事,当我在计算机上本地运行此功能时,它运行正常,没有任何问题。
turns out after starting a thread, you should join them because, once the process quits, the threads die as well. 在启动线程后发现,应该加入它们,因为一旦进程退出,线程也会死亡。 So I did the following
所以我做了以下
import boto3
from threading import Thread
MAX = 999
threads = []
class myThread(Thread):
def __init__(self, bucket_name, objects):
Thread.__init__(self)
self.bucket_name = bucket_name
self.objects = objects
def run(self):
s3 = boto3.client('s3', region_name="us-east-1")
response = s3.delete_objects(Bucket=self.bucket_name, Delete={'Objects':self.objects})
print(response)
def handler(event, context):
keys = event["keys"]
bucket_name = event["bucket"]
objects_to_delete = [1...100,000]
while (len(objects_to_delete) != 0):
t = myThread(bucket_name, objects_to_delete[0:MAX])
threads.append(t)
del objects_to_delete[0:MAX]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
return {
"message": "Success Message."
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.