如何打开流而不是将整个文件加载到pyhton lambda内部的内存中

Question

Hi I am new to lambda and python. I have a use case to read the content of a large file let's say gretaer than 1 GB and just log its content line by line.

I have made a lambda function as below :

import boto3

def lambda_handler(event, context):
    """Read file from s3 on trigger."""
    s3 = boto3.resource('s3')
    file_obj = event['Records'][0]

    bucketname = str(file_obj['s3']['bucket']['name'])
    filename = str(file_obj['s3']['object']['key'])

    iterator = s3.Object(bucketname, filename).get()['Body'].iter_lines()
    for line in iterator:
        print(line)

    return 'Lambda executed successfully.'

But it is not printing anything inside logs.

I think s3.Object(bucketname, filename).get()['Body'] is trying to load the whole file into memory. Is this my understanding is correct? because this is working fine for small files.

If yes, then how can I open a file as a stream without loading it fully inside the memory.

Answer 1

This is work for me

s3 = boto3.resource('s3')
obj = s3.Object(BUCKET, key)
for line in obj.get()['Body']._raw_stream:
    # do something with line