Correct implementation for Kubernetes Liveness and Readiness probes

Question

Given a Python application which polls Kafka topic in an infinite loop and uploads the result to s3 bucket after processing the received Kafka message.

What should be the things to be considered in defining readiness and liveness probes for Kubernetes.

Does it make sense to include in readiness probe:

1. That the s3 buckets exist.
2. That Kafka topic exist.
3. Loop which polls Kafka topic has been initialized.

And the liveness probe only check that the poll loop has not exited.

Is it strictly a bad practice to check such things in readiness probe?

Answer 1

I would not check any of these things in Kubernetes probes. Have your application startup check for them on its own, and if the environment isn't suitable, exit immediately. Your pod will show up in CrashLoopBackOff state, and it will restart a couple of times, but it will be very clear that something is wrong.

There is some possibility that these things will fail while the application is running, but you should be able to notice this. A metrics system like Prometheus can help you notice if most of your S3 requests are failing, for example. If you can check to see if your Kafka listener main loop has exited, you can also just restart it.