AWS Lambda 上的节点、Postgres 出现“连接意外终止”错误

Question

I have a number of Node functions running on AWS Lambda. These functions have been using the Node 8 runtime but AWS sent out an end-of-life notice saying that functions should be upgraded to the latest LTS. With that, I upgraded one on my functions to use Node 12. After being in production for a bit, I'm starting to see a ton of connection terminated unexpectedly errors when querying the database.

Here are the errors that I'm seeing:

1. The connection terminated unexpectedly error
2. And Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed - this seems to happen on the 1st or second invocation after seeing the connection terminated unexpectedly error.

I'm using Knex.js for querying the database. I was running older version of knex and node-postgres and recently upgraded to see if it would resolve the issue, but no luck. Here are the versions of knex and node-postgres that I'm currently running:

"knex": "^0.20.8" "pg": "^7.17.1"

The only change I've made to this particular function is the upgrade to Node 12. I've also tried Node 10, but the same issue persists. Unfortunately, AWS won't let me downgrade to Node 8 to verify that it is indeed an issue. None of my other functions running on Node 8 are experiencing this issue.

I've researched knex , node-postgres and tarn.js (the Knex connection pooling library) to see if any related issues or solutions popped up, but so far, I haven't had any luck.

UPDATE:

Example of a handler. Note that this is happening on many different Lambdas, all running Node 12.

require('../../helpers/knex')

const { Rollbar } = require('@scoutforpets/utils')
const { Email } = require('@scoutforpets/notifications')
const { transaction: tx } = require('objection')
const Invoice = require('../../models/invoice')

// configure rollbar for error logging
const rollbar = Rollbar.configureRollbar(process.env.ROLLBAR_TOKEN)

/**
 *
 * @param {*} event
 */
async function handler (event) {
  const { invoice } = event
  const { id: invoiceId } = invoice

  try {
    return tx(Invoice, async Invoice => {
      // send the receipt
      await Email.Customer.paymentReceipt(invoiceId, true)

      // convert JSON to model
      const i = Invoice.fromJson(invoice)

      // mark the invoice as having been sent
      await i.markAsSent()
    })
  } catch (err) {
    return err
  }
}

module.exports.handler = rollbar.lambdaHandler(handler)

Answer 1

I have a number of Node functions running on AWS Lambda. These functions have been using the Node 8 runtime but AWS sent out an end-of-life notice saying that functions should be upgraded to the latest LTS. With that, I upgraded one on my functions to use Node 12. After being in production for a bit, I'm starting to see a ton of connection terminated unexpectedly errors when querying the database.

Here are the errors that I'm seeing:

1. The connection terminated unexpectedly error
2. And Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed - this seems to happen on the 1st or second invocation after seeing the connection terminated unexpectedly error.

I'm using Knex.js for querying the database. I was running older version of knex and node-postgres and recently upgraded to see if it would resolve the issue, but no luck. Here are the versions of knex and node-postgres that I'm currently running:

"knex": "^0.20.8" "pg": "^7.17.1"

The only change I've made to this particular function is the upgrade to Node 12. I've also tried Node 10, but the same issue persists. Unfortunately, AWS won't let me downgrade to Node 8 to verify that it is indeed an issue. None of my other functions running on Node 8 are experiencing this issue.

I've researched knex , node-postgres and tarn.js (the Knex connection pooling library) to see if any related issues or solutions popped up, but so far, I haven't had any luck.

UPDATE:

Example of a handler. Note that this is happening on many different Lambdas, all running Node 12.

require('../../helpers/knex')

const { Rollbar } = require('@scoutforpets/utils')
const { Email } = require('@scoutforpets/notifications')
const { transaction: tx } = require('objection')
const Invoice = require('../../models/invoice')

// configure rollbar for error logging
const rollbar = Rollbar.configureRollbar(process.env.ROLLBAR_TOKEN)

/**
 *
 * @param {*} event
 */
async function handler (event) {
  const { invoice } = event
  const { id: invoiceId } = invoice

  try {
    return tx(Invoice, async Invoice => {
      // send the receipt
      await Email.Customer.paymentReceipt(invoiceId, true)

      // convert JSON to model
      const i = Invoice.fromJson(invoice)

      // mark the invoice as having been sent
      await i.markAsSent()
    })
  } catch (err) {
    return err
  }
}

module.exports.handler = rollbar.lambdaHandler(handler)

Answer 2

I think you need to set the right connection pooling config.

See the docs here: https://github.com/marcogrcr/sequelize/blob/patch-1/docs/manual/other-topics/aws-lambda.md

const { Sequelize } = require("sequelize");

let sequelize = null;

async function loadSequelize() {
  const sequelize = new Sequelize(/* (...) */, {
    // (...)
    pool: {
      /*
       * Lambda functions process one request at a time but your code may issue multiple queries
       * concurrently. Be wary that `sequelize` has methods that issue 2 queries concurrently
       * (e.g. `Model.findAndCountAll()`). Using a value higher than 1 allows concurrent queries to
       * be executed in parallel rather than serialized. Careful with executing too many queries in
       * parallel per Lambda function execution since that can bring down your database with an
       * excessive number of connections.
       *
       * Ideally you want to choose a `max` number where this holds true:
       * max * EXPECTED_MAX_CONCURRENT_LAMBDA_INVOCATIONS < MAX_ALLOWED_DATABASE_CONNECTIONS * 0.8
       */
      max: 2,
      /*
       * Set this value to 0 so connection pool eviction logic eventually cleans up all connections
       * in the event of a Lambda function timeout.
       */
      min: 0,
      /*
       * Set this value to 0 so connections are eligible for cleanup immediately after they're
       * returned to the pool.
       */
      idle: 0,
      // Choose a small enough value that fails fast if a connection takes too long to be established.
      acquire: 3000,
      /*
       * Ensures the connection pool attempts to be cleaned up automatically on the next Lambda
       * function invocation, if the previous invocation timed out.
       */
      evict: CURRENT_LAMBDA_FUNCTION_TIMEOUT
    }
  });

  // or `sequelize.sync()`
  await sequelize.authenticate();

  return sequelize;
}

module.exports.handler = async function (event, callback) {
  // re-use the sequelize instance across invocations to improve performance
  if (!sequelize) {
    sequelize = await loadSequelize();
  } else {
    // restart connection pool to ensure connections are not re-used across invocations
    sequelize.connectionManager.initPools();

    // restore `getConnection()` if it has been overwritten by `close()`
    if (sequelize.connectionManager.hasOwnProperty("getConnection")) {
      delete sequelize.connectionManager.getConnection;
    }
  }

  try {
    return await doSomethingWithSequelize(sequelize);
  } finally {
    // close any opened connections during the invocation
    // this will wait for any in-progress queries to finish before closing the connections
    await sequelize.connectionManager.close();
  }
};

It's actually for sequelize, not knex, but I'm sure under the hood they work the same way.

Answer 3

I had this problem too, in my case it was cause i tried to connect db in production.

so, I added ssl to Pool, like this:

const pool = new Pool({
    connectionString: connectionString,
    ssl: {rejectUnauthorized: false},
});

Hope it helps you too...