多线程python上的“文本文件繁忙”错误

Question

I have a python script which downloads shell scripts from amazon S3 server and then executes them (each script is about 3GB in size). The function that downloads and executes the file looks like this:

import boto3

def parse_object_key(key):
    key_parts = key.split(':::')
    return key_parts[1]

def process_file(file):
    client = boto3.client('s3')
    node = parse_object_key(file)
    file_path = "/tmp/" + node + "/tmp.sh"
    os.makedirs(file_path)
    client.download_file('category', file, file_path)
    os.chmod(file_path, stat.S_IXUSR)
    os.system(file_path)

The node is unique for each file.

I created a for loop to execute this:

s3 = boto3.resource('s3')
bucket = s3.Bucket('category')
for object in bucket.objects.page_size(count=50):
    process_file(object.key, client)

This works perfectly, but when I try to create a separate thread for each file, I get error:

sh: 1: /path/to/file: Text file busy

The script with threading looks like:

s3 = boto3.resource('s3')
bucket = s3.Bucket('category')
threads = []
for object in bucket.objects.page_size(count=50):
    t = threading.Thread(target=process_file, args=(object.key, client))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

Out of all the threads, exactly one thread succeed and all other fail on "Text file busy error". Can someone help me figure out what I am doing incorrectly?

Answer 1

Boto3 is not thread-safe so you cannot re-use your S3 connection for each download. See here for details of a workaround.