Cloud Run 上的 Nest.js 使用公牛运行计划作业会引发随机连接错误？

Question

I'm trying to perform 3rd party API requests for fetching loads of paginated data (an array of ~16k objects with each object having ~25 key/value pairs), i thought i was getting rate limited but the requests run fine locally and in production as well when triggered manually

but when they run on schedule by bull i get these random connection errors

errorMessage: 'Client network socket disconnected before secure TLS connection was established',
errorStack: 'Error: Client network socket disconnected before secure TLS connection was established\n' +
  '    at connResetException (node:internal/errors:691:14)\n' +
  '    at TLSSocket.onConnectEnd (node:_tls_wrap:1585:19)\n' +
  '    at TLSSocket.emit (node:events:402:35)\n' +
  '    at endReadableNT (node:internal/streams/readable:1343:12)\n' +
  '    at processTicksAndRejections (node:internal/process/task_queues:83:21)'

errorMessage: 'socket hang up',
errorStack: 'Error: socket hang up\n' +
  '    at connResetException (node:internal/errors:691:14)\n' +
  '    at TLSSocket.socketOnEnd (node:_http_client:471:23)\n' +
  '    at TLSSocket.emit (node:events:402:35)\n' +
  '    at endReadableNT (node:internal/streams/readable:1343:12)\n' +
  '    at processTicksAndRejections (node:internal/process/task_queues:83:21)'

errorMessage: 'read ECONNRESET',
errorStack: 'Error: read ECONNRESET\n' +
  '    at TLSWrap.onStreamRead (node:internal/stream_base_commons:220:20)'

errorMessage: 'write ECONNRESET',
errorStack: 'Error: write ECONNRESET\n' +
  '    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:98:16)'

I'm running a Nest.js server on Cloud Run

Here's my Docker configuration

FROM node:16-alpine as build

WORKDIR /app

# Copy and install dependencies first so that they can be cached
COPY package.json .
COPY yarn.lock .
RUN yarn install

# Copy rest of the files and build
COPY . .
RUN yarn build
RUN ls -a

FROM node:16-alpine

WORKDIR /app

# Copy and install dependencies first so that they can be cached
COPY package.json .
RUN yarn install --prod

# Copy files and dist folder from build image
COPY . .
COPY --from=build /app/dist ./dist
RUN ls -a


EXPOSE 3000

CMD ["node", "dist/src/main.js"]

This instance has 4GiB memory, 2 CPUs, request timeout set to 2400 seconds and maximum requests per container to 80, minimum & maximum number of instances are set to 1 and 25 respectively.

And all of the traffic is routed through a VPC connector, it is a f1-micro instance with min instances set to 2 and max 10.

For bull there are 5 different workers setup which are randomly assigned to the job

I wasn't getting these errors on App Engine where this was previously deployed to

Does anyone know what's causing this?

Answer 1

I suggest review the Concurrency value set in Cloud Run. By default each Cloud Run container can receive 80 requests at the same time but you can increase to a maxim of 1000, you can modify this value depending if your code can process parallel requests at the same time, in this link you can find how to set your concurrency value to Cloud Run