Lambda fails to publish to SNS topic sometimes

Question

My AWS Lambda fails to publish messages to an AWS SNS topic every now and then.

2020-03-09T08:02:42.520Z Could not publish on sns with error: NetworkingError: write EPROTO
2020-03-09T08:02:42.789Z Could not publish on sns with error: NetworkingError: Client network socket disconnected before secure TLS connection was established
2020-03-09T08:04:14.797Z Could not publish on sns with error: NetworkingError: connect EPIPE 54.239.55.119:443

I'm using aws-sdk:2.632.0 on Node v12.13.0 . It seems like those issues show up in blocks (like 10 in a row and then not a single issue for an hour).

The code itself is not very sophisticated:

const AWS = require('aws-sdk')
const executionRegion = process.env.AWS_REGION

const sns = new AWS.SNS({
  region: executionRegion
})

async function publishMessage(message, topic) {
  const snsMessage = {
    Message: JSON.stringify(message),
    TopicArn: topic
  }
  try {
    winston.info(`publish sns message ${JSON.stringify(snsMessage)}`);
    return await sns.publish(snsMessage).promise()
  } catch (err) {
    winston.error(`Could not publish message on sns :${err}`)
    return null
  }
}

Any idea what I could try to identify the issue?

Answer 1

Sorry for the necropost, but I had encountered this post and just wrapped up a debugging session with very similar reported behavior. The issue was apparently that SNS was being dispatched too quickly to the AWS SNS endpoint and the JS aws-sdk client was failing as a result. The specific.network error included the following information:

..."errorMessage":"UnknownEndpoint: Inaccessible host: `sns.us-region-1.amazonaws.com' at port `undefined'. This service may not be available in the `us-region-1' region."...
..."errorType":"NetworkingError","errorMessage":"write EPROTO","code":"NetworkingError","message":"write EPROTO","errno":-71,"syscall":"write","region":"us-region-1","hostname":"sns.us-region-1.amazonaws.com"...
..."stack":["Error: write EPROTO"...
...Request.ENOTFOUND_ERROR...

The solution was to serialize the dispatching of the SNS in the JS code. Async dispatch of a couple dozen SNS messages reliably caused the bug on our side.