Kubernetes Rollout 在新 Pod 未完全准备好时丢弃旧 Pod

Question

I'm using the kubectl rollout command to update my deployment. But since my project is a NodeJS Project. The npm run start will take some take(a few seconds before the application is actually running.) But Kubernetes will drop the old pods immediately after the npm run start is executed.

For example,

kubectl logs -f my-app

> my app start
> nest start

The Kubernetes will drop the old pods now. However, it will take another 10 seconds until

Application is running on: http://[::1]:5274

which means my service is actually up.

I'd like to know whether there is a way to modify this like waiting some more time before kubernetes drop the old pods.

My docker file:

FROM node:14 AS builder
# Create app directory
WORKDIR /app
# A wildcard is used to ensure both package.json AND package-lock.json are copied
COPY package*.json ./
COPY prisma ./prisma/
COPY protos ./protos/
COPY tsconfig.build.json ./
COPY tsconfig.json ./
# Install app dependencies
RUN npm install
RUN export NODE_OPTIONS=--max_old_space_size=16384
RUN npm run build
COPY . .
# FROM node:14
# COPY --from=builder /app/node_modules ./node_modules
# COPY --from=builder /app/package*.json ./
# COPY --from=builder /app/dist ./dist
# COPY --from=builder /app/protos ./protos
EXPOSE 5273
CMD ["npm", "run", "start"]

Spec for my kubernetes yaml file:

spec:
  replicas: 4
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: image
        imagePullPolicy: Always
        resources:
          limits:
            memory: "8Gi"
            cpu: "10"
          requests:
            memory: "8Gi"
            cpu: "10"
        livenessProbe:
          httpGet:
            path: /api/Health
            port: 5274
          initialDelaySeconds: 180
          periodSeconds: 80
          timeoutSeconds: 20
          failureThreshold: 2
        ports:
        - containerPort: 5274
        - containerPort: 5900

Answer 1

Use a startup probe on your container. https://docs.openshift.com/container-platform/4.11/applications/application-health.html . Pods don't count as "ready" until all of their containers have passed their startup (and readiness) checks.

And during a deployment the scheduler counts non-ready pods as "unavailable" for things like the "maxUnavailable" setting of the deployment. Thus the scheduler won't keep shutting down working pods until new pods are ready for traffic. ( https://docs.openshift.com/container-platform/4.11/applications/deployments/deployment-strategies.html )

As an additional benefit, services won't route traffic to non-ready pods, so they won't receive any traffic until the containers have passed their startup probes.