AWS Lambda和S3：将s3对象路径传递到图像处理功能

Question

My intention is to have a large image stored on my S3 server and then get a lambda function to read/process the file and save the resulting output(s). I'm using a package called python-bioformats to work with a proprietary image file (which is basically a whole bunch of tiffs stacked together). When I use

def lambda_handler(event, context):

    import boto3

    key = event['Records'][0]['s3']['object']['key'].encode("utf-8")
    bucket = 'bucketname'

    s3 = boto3.resource('s3')
    imageobj = s3.Object(bucket, key).get()['Body'].read()

    bioformats.get_omexml_metadata(imageobj)

I have a feeling that the lambda function tries to download the entire file (5GB) when making imageobj. Is there a way I can just get the second function (which takes a filepath as argument) to refer to the s3 object in a filepath-like manner? I'd also like to not expose the s3 bucket/object publicly, so doing this server-side would be ideal.

Answer 1

If your bioformats.get_omexml_metadata() function requires a filepath as an argument, then you will need to have the object downloaded before calling the function.

This could be a problem in an AWS Lambda function because there is a 500MB limit on available disk space (and only in /tmp/ ).

If the data can instead be processed as a stream, you could read the data as it is required without saving to disk first. However, the python-bioformats documentation does not show this as an option. In fact, I would be surprised if your above code works, given that it is expecting a path while imageobj is the contents of the file.