AWS Lambda - Mongoose connection timeout

Question

I'm trying to reuse the mongodb connection for AWS Lambdas, this is my connection function:

const openMongodbConnection = ({ config }) => (
  process.mongoConnection ? Promise.resolve() : mongoose.connect(config.MONGO_URL, {
    bufferCommands: false,
    bufferMaxEntries: 0,
    keepAlive: true,
  })

  .then((connection) => {
    process.mongoConnection = connection;
    return Promise.resolve();
  }, err => (
    Promise.reject({
      statusCode: 500,
      errorCode: [{
        code: 'DatabaseError',
        description: `unable to connect to mongo db: ${err} ${JSON.stringify(config)}`,
      }],
    })
  ))
);

Database is within an AWS VPC and doesn't have external access. Cold start works perfectly, but sometimes I get a timeout error like this:

2018-10-03T18:36:06.984Z    7ab97df7-c739-11e8-89bf-87260b172585    MongoNetworkError: connection 2 to some.ip.from.server:27017 timed out
at Socket.<anonymous> (/var/task/node_modules/mongodb-core/lib/connection/connection.js:258:7)
at Socket.g (events.js:292:16)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at Socket._onTimeout (net.js:338:8)
at ontimeout (timers.js:386:14)
at tryOnTimeout (timers.js:250:5)
at Timer.listOnTimeout (timers.js:214:5)

I also have context.callbackWaitsForEmptyEventLoop = false; right on the start of the function.

This error is completely random, sometimes work and sometimes doesn't, I'm thinking on opening and closing the connection everytime a new request comes up, but I know this will reduce the performance of lambdas and also will increase the I/O operations from the mongodb server.

Help will be greatly appreciated.

Answer 1

Well, I came here looking for an answer to this myself.. anyway, the ONLY answer we have found so far is to open and close the DB connection each time the lambda is invoked.

Apparently, when the lambda isn't working, AWS suspends the process which means the socket looks dead to the mongodb server, so it drops the connection because it thinks it's a dead client. Thus trying to keep the socket around between lambda invokes leads to this timeout quite often.

Answer 2

The problem is when the lambda function is trigger from AWS Fargate.

When AWS uses IAMRole:PassRole it brakes the VPC/Subnet configuration, thus making the lambda function not able to connect to the mongo database.

Answer 3

We have experienced the same issue. it turns out our problem was mainly with making the connection outside the handler function .connection was being opened without the handler function being actually called and only by requiring lambda.js file.

The issue has vanished when we moved the connection code inside handler function:

// lambda.js

// DO NOT open mongodb connection here


exports.handler = async (event, context) => {
    // open the mongo connection here
    // openMongodbConnection()
}

I am not exactly sure how could this cause an issue but it is probably has something to do with the sequence to which subnet configuration will be applied.