如何在终止Pod的同时确定kubernetes的宽限期？

Question

we'd like to scale in some of the running instances on which many kubernetes pods are running. So, we are going to gracefully stop the pods by using graceful period according to the official document termination-of-pods . I have read many blog posts and official document, they all tells how to gracefully terminate pod with graceful period . But they do not say how to determine how long the graceful period would be better.

Let's say, for example, a container in a pod may serves for thousands of requests in a time period and it will spend more than 30s to complete all request. I think in this case it would be a bad idea to set graceful period to 30s, because some of the request would be lost. However, when the user load is down and the same container in the same pod serves for only dozens of request in other time period and it only spend 5s to complete all, in this case 30s for graceful period would be too long.

That's my consideration. So, my question is as follows. 1. Is there any best practice to determine how long the graceful period is better? 2. Is there any approach to check if the processing request is completed in a container and then gracefully terminate pod? 3. Can I extend the initial graceful period after sending the termination command to a pod?

Thanks in advance.

Answer 1

The best way to determine the ideal graceful period is by observability. Put your service under a realistic production load and measure. This is highly project specific!

If the process with PID 1 exits before the graceful period your container will be marked as Terminated before the end of the graceful period, so it's worth setting a value slightly higher than what you would expect under normal circumstances.

You might be interested in letting your containers write arbitrary information when they terminate. Kubernetes has a feature called Termination messages you might want to look into.