如何將文件列表從 web 直接上傳到 s3 存儲桶

Question

我的 s3 存儲桶中有一個包含 url 的文件。 我想使用 python lambda function 將 url 文件上傳到 s3 存儲桶。

例如我上傳到 s3 的文件包含：http://... http://...

每行對應一個要上傳到 s3 的文件。

這是代碼：

import json
import urllib.parse
import boto3
import requests
import os
from gzip import GzipFile
from io import TextIOWrapper
import requests

print('Loading functions')

s3 = boto3.client('s3')


def get_file_seqs(response):
    try:
        size = response['ResponseMetadata']['HTTPHeaders']['content-length']
        print("[+] Size retrieved")
        return size
    except:
        print("[-] Size can not be retrieved")

def lambda_handler(event, context):
    # Defining bucket objects
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'],   encoding='utf-8')
    #get file from s3
    print('[+] Getting file from S3 bucket')
    response = s3.get_object(Bucket=bucket, Key=key)

    try:
        #checking file size
        print('[+] Checking file size')
        file_size =  get_file_seqs(response)
        if file_size == 0:
            print('File size is equal to 0')
            return False
        else:
            #create new directories
            print('[+] Creating new directories')
            bucket_name = "triggersnextflow"
            directories = ['backups/sample/', 'backups/control/']
            #loop to create new dirs
            for dirs in directories:
                s3.put_object(Bucket = bucket_name, Key = dirs, Body = '')
        
        #NOW I WOULD LIKE TO DOWNLOAD THE FILES FROM THE URLS INSIDE S3 OBJECT


     
        #return true
        return True
        
    except Exception as e:
        print(e)
        print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
        raise e

Answer 1

將 S3 object 下載到文件中：

import boto3
s3 = boto3.resource('s3')
s3.meta.client.download_file('mybucket', 'hello.txt', '/tmp/hello.txt')

您會在這里找到大量的信息資源： https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.download_file