AWS Lambda 函数可以直接处理 s3 上的文件还是需要移动到 /tmp/？

Question

I am trying to write an AWS lambda function in Python to collect a bunch of csv files from an s3 bucket, concatenate them, drop the duplicates and write the result back to s3. The files I want to read are stored with a prefix/in a folder on s3. Currently I am trying to read the files one by one using the following apporach:

resp = s3.list_objects_v2(Bucket='mybucket')
#getting all objects in the bucket in a list
for obj in resp['Contents']:
    keys.append(obj['Key'])
#filtering those that are parsed entries
files = [k[6:] for k in keys if 'links/links' in k]
#reading into combined list
for file in files:
    with open(file, 'r') as csvfile:
        reader = csv.reader(csvfile)
        links = links + list(reader)

Currently I am getting the following error:

{
  "errorMessage": "[Errno 2] No such file or directory: 'links2020-02-27 14:59:49.933074.csv'",
  "errorType": "FileNotFoundError",
  "stackTrace": [
    "  File \"/var/task/handler.py\", line 21, in concatenatelinks\n    with open(file, 'r') as csvfile:\n"
  ]
}

In an earlier version, I didn't slice the filenames, which caused the same error. So do I need to load all files into /tmp/ with something like s3.meta.client.upload_file('/tmp/' + str(filename), bucket, 'fusedlinks/' + str(filename)) to make them accessible to the lamda function or is there a more elegant solution to this?

Answer 1

From the error it seems that filename convention is incorrect: links2020-02-27 14:59:49.933074.csv . You probably need to escape the " whitespace " while reading the file through boto3 client.

But to read the file there are two options, I personally prefer Option 2 (but depends on memory usage):

1. One to use filesystem as /tmp

You can refer the sample example as mentioned on AWS Documentation

Also AWS Lambda provides /tmp size of 512 MB at the moment, you will need to find a different solution if total size of all files is more than 512 MB. Refer AWS Lambda Limits

2. Second option is to use In-memory buffer . You can use: Python's BytesIo . Example below:

    def load_from_s3(bucket, path):
        s3_resource = boto3.resource('s3')
        with BytesIO() as data:
            s3_resource.Bucket(bucket).download_fileobj(path, data)
            data.seek(0)
            # Do something with your data in file