[英]Prometheus in kubernetes have too many unhealthy targets
First, My kubernetes cluster is based on Bare-metal environment.首先,我的 kubernetes 集群是基于裸机环境的。
cluster info:集群信息:
k get no -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k-master Ready master 142d v1.18.10 192.168.6.211 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://18.9.9
k-node-1 Ready <none> 142d v1.18.10 192.168.6.212 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://18.9.9
k-node-2 Ready <none> 142d v1.18.10 192.168.6.213 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://18.9.9
When i install Prometheus in cluster to monitor k8s, My step:当我在集群中安装 Prometheus 来监控 k8s 时,我的步骤:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitor
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-conf
namespace: monitor
data:
prometheus.yml: |
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s #
scrape_configs:
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- job_name: 'kubernetes-service'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: service
- job_name: 'kubernetes-endpoints'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: endpoints
- job_name: 'kubernetes-ingress'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: ingress
- job_name: 'kubernetes-pods'
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-server
namespace: monitor
spec:
selector:
matchLabels:
app: prometheus-server
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus-server
image: prom/prometheus
volumeMounts:
- name: conf
mountPath: "/etc/prometheus"
readOnly: true
ports:
- containerPort: 9090
serviceAccountName: prometheus
volumes:
- name: conf
configMap:
name: prometheus-conf
Anyway, service and ingress of prometheus related has already created, and i can access prometheus webUI, but so many Unhealthy Targets in Service Discovery page.无论如何,prometheus相关的服务和入口已经创建,我可以访问prometheus webUI,但是服务发现页面中有很多不健康的目标。
For kubernetes-nodes details: server returned HTTP status 403 Forbidden对于 kubernetes 节点的详细信息:服务器返回 HTTP 状态 403 Forbidden
I don't know how to fix it and other more.我不知道如何解决它和其他更多。 Can anybody teach me?
有人可以教我吗? thanks!
谢谢!
You can check the server side certification file for kubelet:您可以检查 kubelet 的服务器端认证文件:
# openssl x509 -text -noout -in /var/lib/kubelet/pki/kubelet.crt
Maybe another file name.也许另一个文件名。 and check the SAN part of the output, like this:
并检查 output 的 SAN 部分,如下所示:
X509v3 Subject Alternative Name:
DNS:host.yourdomain.com
If there is only the node name get displayed, then you access it through ip will not succeed.如果只显示节点名,那么通过ip访问是不会成功的。 In this way, either probe the node through api-server, or regenerate the certification files with ip in the SAN field.
这样,要么通过api-server探测节点,要么重新生成SAN字段中带有ip的认证文件。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.