Upload files to S3 from multipart/form-data in AWS Lambda (Python)

Question

I'm calling this Lambda function via API Gateway. My issue is that the image file is malformed, meaning that it does not open.

import boto3
import json

def lambda_handler(event, context):
    print(event)

    # removing all the data around the packet
    # this also results in a malformed png
    start = '<?xpacket end="r"?>'
    end = '\r\n------'
    content = str(event['body'])
    content = content[content.index(start) + len(start):content.index(end)].encode('utf-8')
    
    bucket_name = "bucket-name"
    file_name = "hello1.png"
    lambda_path = "/tmp/" + file_name
    s3_path = file_name

    s3 = boto3.resource("s3")
    s3.Bucket(bucket_name).put_object(Key=s3_path, Body=content)
    
    return {
        'statusCode': 200,
        'headers': {
            'Access-Control-Allow-Origin': '*',
        },
        'body': json.dumps(event)
    }

Answer 1

Lambda has payload limit of 6mb for synchronous call and 256KB for async call.

https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Also api gateway has limit of 10MB for RESTfull APIs and 128KB for socket message

https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

It may be the primary reason why some part of file is uploaded, some not.

Even if you do not hit those limits with smaller file sizes, you pay for lambda execution while uploading. It is just a waste of lambda's time.

Also there may be config on api gateway to modify payload while pushing it to lambda. Make sure there is no active template which would convert the request before hitting lambda and check if use lambda as proxy is checked at gateway-api dashboard for this resource.

To upload to S3 better use Pre-Signed URL for an Amazon S3 PUT Operation:

https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html