AWS Lambda 上传到 S3 超时

Question

I'm fairly new to AWS Lambda and I'm facing a problem. I'm developing in Node and I'm downloading JSON file (quite big ones 12-25MB) from my S3, doing some processing and trying to upload the result in form of JSON back to S3.

I was following this tutorial: https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html which does essentially the same thing.

My problem is that my lambda always times out when calling JSON.stringify on an object that I get as a result of my data processing and thus the file never gets uploaded back to S3. The result of my processing is also a quite big object but still smaller as the input (cca. 10MB).
I already tried to use fast-json-stringify library but I ended up with the same result.

Does anyone know if there is any other faster way to convert an object to string, buffer or stream?

UPDATE

Here is my code:

async.waterfall(
  [
    (next) => {
      s3.getObject({
        Bucket: bucketName,
        Key: keyDownload
      }, next);
    },
    (response, next) => {
      next(null, processData(JSON.parse(response.Body)));
    },
    (data, next) => {
      s3.putObject({
        Bucket: bucketName,
        Key: keyUpload,
        Body: JSON.stringify(data)
      }, next);
    }
  ], 
  (err) => {
    if (err) {
      callback(err);
    } else {
      callback();
    }
  }
);

Answer 1

Did you try S3 multi-part upload, I would suggest that's a place to look into -

https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

Answer 2

• Just a random thought after looking at the example you are following.

• if you are using the aws cli to create the lambda function just wondering what is the --timeout value that you have given. The example says --timeout 10 which is 10 sec which is definitely small time span.

• You can check/change it on the aws console as well and try and increase it to say 10 min ie 600 sec (this is just for testing if this is the problem or not later on you can get down to some appropriate value after evaluation).

• The timeout is optional value while creating via aws cli. Thus the default is 3 seconds. The maximum allowed value is 900 seconds ie 15 min.

• Definitely multi-part upload will be helpful.