Could not reach Cloud Firestore backend on CloudRun with Node.js

Question

I've an Angular Universal application deployed to CloudRun (running Node.js on the server) and connecting to a Firestore DB. It's been working successfully for the last 2 years.

I started getting the Could not reach Cloud Firestore backend on CloudRun error since 2 days ago (I've run a query on Cloud logger for the last 2 months, and only last 2 days started being problematic).

Some things to consider:

• A version to prod was deployed the day this issue started; nothing in the code should be related to this issue, an non node_module was updated ( package-lock didn't change)
• The app is runs on a Docker image; some dependency might have changed, and probably something got changed here and initiated the issue? (I'm not a docker/linux expert)
• The issue ONLY happens intermittently ; I've some health-checks in place, and I receive several alerts during the day, but sometimes it happens with a window delay of 30 minutes, to 2 hours, or more than 6 hours.
• The offline issue seems to never last for more than 10 minutes
• The issue becomes more evident when there is only 1 container instance running; when multiple instances are active, most of them works without any problem

It's not clear if the issue is from CloudRun non being able to connect to Firestore (eg due to lack of internet/egress-networking (?)), or if Firestore is somehow refusing the connection.

Has someone faced a similar issue, or can give me suggestions on how to approach/solve this incident?

Thank you very much in advance

Answer 1

Posting the updated and solving the issue in case it helps anybody else...

I've finally solved the issue by "rolling back" the Node alpine version used by my Docker image.

The Docker image I was using was created using FROM node:alpine , which points to the newest stable version of Node.js.

Apparently, that last version introduced the bug; I've no idea what was causing the issue, but it started after upgrading to Node.js v18.8.0 .

I've rolled back to a previous version ( FROM node:current-alpine3.14 which runs Node.js v18.3.0 ) and the error finally disappeared.