Kubernetes ClusterIP service initial delay or liveness

Question

I have a Kubernetes deployment on GCP and a ClusterIP service to discover pods in this deployment. The deployment contains multiple replica set pods which come and go based on our horizontal pod scalar configuration (based on CPU Utilization).

Now, when a new replica set pod is created, it takes some time for the application to start servicing. But the ClusterIP already starts distributing requests to new replica set pod before the application is ready, which causes the requests to be not serviced.

ClusterIP service yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: service-name
    tier: backend
    environment: "dev"
    creator: internal
  name: service-name
spec:
  clusterIP: None
  ports:
  - name: https
    protocol: TCP
    port: 7070
    targetPort: 7070
  selector:
    app: dep-name
    tier: "backend"
    environment: "dev"
    creator: "ME"
  type: ClusterIP

How can the ClusterIP be told to start distributing requests to the new pod after the application starts? Can there be any initial delay or liveness probe set for this purpose?

Answer 1

Kubernetes provides readiness probe for it. With readiness probes, Kubernetes will not send traffic to a pod until the probe is successful. When updating a deployment, it will also leave old replica(s) running until probes have been successful on new replica. That means that if your new pods are broken in some way, they'll never see traffic, your old pods will continue to serve all traffic for the deployment.

You need to update the deployment file with following readiness probe:

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

If your application have http probe then you can set readiness probe in HTTP mode as well.

For more information how can you use readiness probe refer:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-readiness-probes

Answer 2

You should have a readiness probe as defined in the documentation at https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-readiness-probes .

As defined in the documentation you should be able to configure using initialDelaySeconds and periodSeconds .

Your current behavior is probably because the service load balancer has seen that all the containers in the pod are started. You can define your readyness checks like the example below picked from documentation.

kind: Pod
metadata:
  name: goproxy
  labels:
    app: goproxy
spec:
  containers:
  - name: goproxy
    image: k8s.gcr.io/goproxy:0.1
    ports:
    - containerPort: 8080
    readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20