Kubernetes：服务器停止后 Kafka pod 关闭

Question

I am running a Kafka pod in Kubernetes with Rancher. I am using the confluent image and it is connecting properly to zookeeper. I am using the stable helm chart with the Kafka confluent image 5.3.1. I also added SSL encryption into Helm using this page It is starting properly then it shutdown abruptly and pod restart. I am getting this error on the log.

[2019-11-15 19:41:49,943] INFO Terminating process due to signal SIGTERM (org.apache.kafka.common.utils.LoggingSignalHandler) [2019-11-15 19:41:49,945] INFO Shutting down SupportedServerStartable (io.confluent.support.metrics.SupportedServerStartable)

What is the SIGTERM error in Kafka pods? How to fix it?

Thank you

Answer 1

This is liveness and readiness problem. When kafka attempt to read topic snapshots liveness ping to kafka and it can't respond. So k8 shutdown kafka. Remove liveness and readiness

Answer 2

As mentioned by @Hamzatli, it is about liveness and readiness. K8s thinks that your pod is hitting a timeout issue and sends the SIGTERM to the pod for it to shutdown.

In your Helm Chart's values.yaml, there should be an option about liveness and/or readiness. Increase the initialDelaySeconds to a higher time, which you think is enough time for the Kafka to be brought up, so that k8s doesn't send the shutdown signal too quickly during initial startup.

livenessProbe:
  enabled: true
  initialDelaySeconds: 60   # 60 seconds delay for the pod to start liveness probe
  timeoutSeconds: 5
readinessProbe:
  enabled: true
  initialDelaySeconds: 60   # 60 seconds delay for pod to start readiness probe
  timeoutSeconds: 5

You can read more about this here .