How to implement custom logic for kubernetes pod termination?

Question

I have a pod that is stateful (handles phone calls from an IVR) and due to that, performing updates or having an HPA perform pod termination can protentional cause dropped phone calls. I could extend the terminationGracePeriodSeconds to something large like 10 minutes, but I do not want to extend this that far if there are no phone calls coming through this pod.

Is there some way to query the pod via rest endpoint to verify how many phone calls are active? I would like to terminate the pod fast, but also extend it if it still had ongoing conversations.

Some details that may be pertinent:

• K8s: v1.20
• Language: NodeJS v16

Answer 1

You can use preStop handler, see https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/

Let's say your /calls endpoint returns number of current calls, you can do something like this, to query this endpoint each 5 seconds, until call number reaches zero

  containers:
  - name: your container
    # . . . 
    lifecycle:
      preStop:
        exec:
          command: ["/bin/bash","-c","while [ \"$(curl localhost/calls)\" != '0' ]; do echo 'there are active calls'; sleep 5; done; echo 'no active calls, exiting'; exit 0" ]

this requires curl to be present in your image, and you may also want to redirect stderr to /dev/null and/or add other failsafes, but should be enough as a proof of concept