AWS Lambda - Nodejs: Allocation failed - JavaScript heap out of memory

Question

I am using Lambda (nodeJS) to query noSQL data (dynamo db).

Let's say I have table "student" in DynamoDB, and I have an API which return list of students for a specific class (class_id). (I used "query" )

As I know, the dynamo paginates the result, so my API works like below:

• {class_id : 'xxxx'} => return 1st list of students
• {class_id : 'xxxx', last_evaluated_key: { ....} => return next list of student (if LastEvaluatedKey exists)

My lambda code :

exports.handler = function(e, ctx, callback) {
    var rp = require('request-promise');

    var students = [];

    var hasMore = true;
    var params = {
        class_id: e.class_id
    }

    while (hasMore) {
        var options = {
            method: 'POST',
            uri: 'https://xxxxxx.eu-west-1.amazonaws.com/dynamodliblightdm-gobject-1-0:amd64liblightdm-gobject-1-0:amd64b/getStudents',
            body: params,
            json: true // Automatically stringifies the body to JSON
        };

        rp(options)
            .then(function(repos) {
                console.log('count: ' + repos.Count);
                students.push(repos.Items);

                if (repos.hasOwnProperty("LastEvaluatedKey")) {
                    params['last_evaluated_key'] = repos.LastEvaluatedKey;
                } else {
                    hasMore = false;
                }

            })
            .catch(function(err) {
                console.log('Error', err);
            });
    }


    callback(null, 'done.');
}

I got error:

42676 ms: Mark-sweep 804.1 (954.3) -> 802.7 (954.3) MB, 1803.0 / 0.0 ms (+ 246.3 ms in 32 steps since start of marking, biggest step 35.7 ms) [allocation failure] [GC in old space requested]. 44415 ms: Mark-sweep 802.7 (954.3) -> 802.7 (954.3) MB, 1738.6 / 0.0 ms [allocation failure] [GC in old space requested]. 46318 ms: Mark-sweep 802.7 (954.3) -> 809.5 (859.3) MB, 1902.9 / 0.0 ms [last resort gc]. 48184 ms: Mark-sweep 809.5 (859.3) -> 816.4 (858.3) MB, 1865.7 / 0.0 ms [last resort gc]. <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x322e8723fa99 2: new constructor(aka Multipart) [/var/task/lambda-func/node_modules/request/lib/multipart.js:~8] [pc=0x1b47df3f5f98] (this=0x1175e583149 ,request=0x1175e582fa9 ) 4: new constructor(aka Request) [/var/task/lambda-func/node_modules/request/request.js:125] [pc=0x1b47df4df3e6] (this=0x1175e... FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [/var/lang/bin/node] 2: 0x55d79ff0b302 [/var/lang/bin/node] 3: v8::Utils::ReportApiFailure(char const*, char const*) [/var/lang/bin/node] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/var/lang/bin/node] 5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/var/lang/bin/node] 6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/var/lang/bin/node] 7: 0x1b47df2062bf

Any suggestion is appreciated.

Answer 1

Suggestion to get all the students.

const fetch  = (lastEvaluatedKey)  => {
  return rp().then((res) => {
    students = students.concat(res.moreStudents);
    if(res.shouldKeepFetching) {
      return fetch(res.lastKey);
    }

    return Promise.resolve();
  })
}

fetch().then(() => {
  //you fetched them all
})

If there are a shitload of students this might get you with another out of memory problem. Another thing you can do, but I don't know if lambda allows it yet, is to use a for loop with async/await