I have configured a spring-boot pod and configured the liveness
and readiness
probes. When I start the pod, the describe
command is showing the below output.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 92s default-scheduler Successfully assigned pradeep-ns/order-microservice-rs-8tqrv to pool-h4jq5h014-ukl3l
Normal Pulled 43s (x2 over 91s) kubelet Container image "classpathio/order-microservice:latest" already present on machine
Normal Created 43s (x2 over 91s) kubelet Created container order-microservice
Normal Started 43s (x2 over 91s) kubelet Started container order-microservice
Warning Unhealthy 12s (x6 over 72s) kubelet Liveness probe failed: Get "http://10.244.0.206:8222/actuator/health/liveness": dial tcp 10.244.0.206:8222: connect: connection refused
Normal Killing 12s (x2 over 52s) kubelet Container order-microservice failed liveness probe, will be restarted
Warning Unhealthy 2s (x8 over 72s) kubelet Readiness probe failed: Get "http://10.244.0.206:8222/actuator/health/readiness": dial tcp 10.244.0.206:8222: connect: connection refused
The pod definition is like below
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: order-microservice-rs
labels:
app: order-microservice
spec:
replicas: 1
selector:
matchLabels:
app: order-microservice
template:
metadata:
name: order-microservice
labels:
app: order-microservice
spec:
containers:
- name: order-microservice
image: classpathio/order-microservice:latest
imagePullPolicy: IfNotPresent
env:
- name: SPRING_PROFILES_ACTIVE
value: dev
- name: SPRING_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: username
name: db-credentials
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: db-credentials
volumeMounts:
- name: app-config
mountPath: /app/config
- name: app-logs
mountPath: /var/log
livenessProbe:
httpGet:
port: 8222
path: /actuator/health/liveness
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
port: 8222
path: /actuator/health/readiness
initialDelaySeconds: 10
periodSeconds: 10
resources:
requests:
memory: "550Mi"
cpu: "500m"
limits:
memory: "550Mi"
cpu: "750m"
volumes:
- name: app-config
configMap:
name: order-microservice-config
- name: app-logs
emptyDir: {}
restartPolicy: Always
If I disable the liveness
and readiness
probe in the replica-set
manifest and I exec
into the pod, I am getting a valid response when invoking http://localhost:8222/actuator/health/liveness
and http://localhost:8222/actuator/health/readiness
endpoint. Why is my pod restarting and failing when invoking the readiness
and liveness
endpoint with Kubernetes. Where am I going wrong?
Update If I remove the resource
section, the pods are running but when added the resource
parameters, the probes
are failing.
When you limit the container / spring application to 0.5 cores (500 millicores) the startup probably takes longer than the given liveness probe thresholds.
You can either increase them, or use a startupProbe with more relaxed settings (fe failureThreshold 10). You can reduce the period for the liveness probe in that case and get faster feedback after a successful container start was detected.
Your pod config only give 0.5 Core of CPU, and your check time was too short. The spring boot start may take a long time more than 10 seconds according your server CPU performance. This is my config of spring boot pod may give you a point.
"livenessProbe": {
"httpGet": {
"path": "/actuator/liveness",
"port": 11032,
"scheme": "HTTP"
},
"initialDelaySeconds": 90,
"timeoutSeconds": 30,
"periodSeconds": 30,
"successThreshold": 1,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/actuator/health",
"port": 11032,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 30,
"periodSeconds": 30,
"successThreshold": 1,
"failureThreshold": 3
},
and I did not limit the CPU and memory resource, if you limit the CPU, it will take more time. Hop this could help you.
When you are trying the request against your localhost
, and it works, it is not a guarantee that it is going to work on other network interfaces. Kubelet is a node agent, so the request is going to your eth0
, or equivalent, not your localhost
.
You can check it by making the request from another pod to your pod's IP address, or the service backing it up.
Probably you are making your application to serve on localhost
, while you have to make it serve on 0.0.0.0
, or eth0
.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.