如何从S3存储桶读取大型XML文件，然后使用AWS Lambda将其用作HTTP请求正文

Question

I'm writing an AWS Lambda, where I need to use the content of an XML file from S3 bucket to make an HTTP PUT request. This lambda should be triggered whenever there's a file upload in the given S3 bucket. Following is my code in Node.js:

exports.handler = (event, context, callback) => {
    var s3 = new AWS.S3();
    var sourceBucket = event.Records[0].s3.bucket.name;
    const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
    const params = {
        Bucket: sourceBucket,
        Key: key
    };
    s3.getObject(params, function(err, data) {
        if (err)
            console.log(err, err.stack); // an error occurred
        else {
            var body = data.Body.toString();
            const params2 = {
                url: 'http://url:port/PutFile',
                body: body
            };
            req.put(params2, function(err, res, body) {
                if(err){
                    console.log('------error------', err);
                } else{
                    console.log('------success--------', body);
                }
            });
            req('http://url:port/ResetData', function (error, response, body) {
                //Check for error
                if(error){
                    return console.log('Error:', error);
                }

                //Check for right status code
                if(response.statusCode !== 200){
                    return console.log('Invalid Status Code Returned:', response.statusCode);
                }
                    console.log(body); 

            });
    });
};

My code is working fine and making required calls for a dummy XML file (~3KB). But for original requirement, with file size being more than 10MB, its returning following exception:

RequestId: <someId> Error: Runtime exited with error: signal: killed

I've tried increasing the lambda timeout to 10 minutes and I'm still getting the same error. How should I resolve this?

Answer 1

The issue is likely your lambda is running out of memory and aws killed execution. If you look at the line above this line:

...Error: Runtime exited with error: signal: killed

you will see the line below (boldface is mine):

REPORT RequestId: XXXX-XXXXX-XXXXX Duration: XXXX.XX ms Billed Duration: XXXXX ms Memory Size: 128 MB Max Memory Used: 129 MB

Notice you've used more memory than the size you allocated to the lambda. Increase the limit & try again.