堆栈内存溢出
  繁体   English   中英

Kubernetes 1.14.2 HA Master NGINX负载均衡器log.go:172] http：来自192.168.5.32:43148的TLS握手错误：远程错误：tls：错误证书

[英]Kubernetes 1.14.2 HA Master NGINX load balancer log.go:172] http: TLS handshake error from 192.168.5.32:43148: remote error: tls: bad certificate

 原文  2019-05-26 05:12:22  6   3    ssl/ kubernetes/ handshake

问题描述

这让我发疯，我不是Kubernetes专家，但我也不是新手。

我已经尝试了三天未能成功通过这个问题，但我不能和我结束我的绳索。

将证书从（kube-apiserver-1：/ etc / kubernetes / pki / *）复制到我的桌面后，我可以从桌面查询群集。 

$ kubectl -n kube-system get nodes
NAME               STATUS   ROLES    AGE   VERSION
kube-apiserver-1   Ready    master   71m   v1.14.2

当我查询kube-system pod时，Kubernetes集群看起来很健康： 

$ kubectl -n kube-system get pods
NAME                                       READY   STATUS    RESTARTS   AGE
coredns-fb8b8dccf-6c85q                    1/1     Running   3          65m
coredns-fb8b8dccf-qwxlp                    1/1     Running   3          65m
kube-apiserver-kube-apiserver-1            1/1     Running   2          72m
kube-controller-manager-kube-apiserver-1   1/1     Running   2          72m
kube-flannel-ds-amd64-phntk                1/1     Running   2          62m
kube-proxy-swxrz                           1/1     Running   2          65m
kube-scheduler-kube-apiserver-1            1/1     Running   1          54m

但是当我查询api kubelet时： 

$ kubectl -n kube-system logs kube-apiserver-kube-apiserver-1 
...
I0526 04:33:51.523828       1 log.go:172] http: TLS handshake error from 192.168.5.32:43122: remote error: tls: bad certificate
I0526 04:33:51.537258       1 log.go:172] http: TLS handshake error from 192.168.5.32:43124: remote error: tls: bad certificate
I0526 04:33:51.540617       1 log.go:172] http: TLS handshake error from 192.168.5.32:43126: remote error: tls: bad certificate
I0526 04:33:52.333817       1 log.go:172] http: TLS handshake error from 192.168.5.32:43130: remote error: tls: bad certificate
I0526 04:33:52.334354       1 log.go:172] http: TLS handshake error from 192.168.5.32:43128: remote error: tls: bad certificate
I0526 04:33:52.335570       1 log.go:172] http: TLS handshake error from 192.168.5.32:43132: remote error: tls: bad certificate
I0526 04:33:52.336703       1 log.go:172] http: TLS handshake error from 192.168.5.32:43134: remote error: tls: bad certificate
I0526 04:33:52.338792       1 log.go:172] http: TLS handshake error from 192.168.5.32:43136: remote error: tls: bad certificate
I0526 04:33:52.391557       1 log.go:172] http: TLS handshake error from 192.168.5.32:43138: remote error: tls: bad certificate
I0526 04:33:52.396566       1 log.go:172] http: TLS handshake error from 192.168.5.32:43140: remote error: tls: bad certificate
I0526 04:33:52.519666       1 log.go:172] http: TLS handshake error from 192.168.5.32:43142: remote error: tls: bad certificate
I0526 04:33:52.524702       1 log.go:172] http: TLS handshake error from 192.168.5.32:43144: remote error: tls: bad certificate
I0526 04:33:52.537127       1 log.go:172] http: TLS handshake error from 192.168.5.32:43146: remote error: tls: bad certificate
I0526 04:33:52.550177       1 log.go:172] http: TLS handshake error from 192.168.5.32:43150: remote error: tls: bad certificate
I0526 04:33:52.550613       1 log.go:172] http: TLS handshake error from 192.168.5.32:43148: remote error: tls: bad certificate

在NGINX负载均衡器（IP：192.168.5.32）上，我已按照Kubernetes文档中的说明配置了TCP passthrough选项： 

upstream kubernetes-api-cluster {
   server 192.168.5.19:6443;
   server 192.168.5.29:6443;
}
server {
   listen 6443;
   ssl_certificate /etc/nginx/ssl/kube-apiserver.pem;
   ssl_certificate_key /etc/nginx/ssl/private/kube-apiserver.key;
   ssl_prefer_server_ciphers on;
   ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
   ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
   proxy_pass kubernetes-api-cluster;
}

我可以直接从NGINX LB（IP：192.168.5.32）查询API服务器： 

$ curl -v https://192.168.5.29:6443
* Rebuilt URL to: https://192.168.5.29:6443/
*   Trying 192.168.5.29...
* TCP_NODELAY set
* Connected to 192.168.5.29 (192.168.5.29) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: May 26 03:39:36 2019 GMT
*  expire date: May 25 03:39:36 2020 GMT
*  subjectAltName: host "192.168.5.29" matched cert's IP address!
*  issuer: CN=kubernetes
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55840f1d9900)
> GET / HTTP/2
> Host: 192.168.5.29:6443
> User-Agent: curl/7.58.0
> Accept: */*

我还可以使用文档中指定的api的DNS条目来查询api： 

curl -v https://kube-apiserver.mydomain.com:6443
* Rebuilt URL to: https://kube-apiserver.mydomain.com:6443/
*   Trying 10.50.1.50...
* TCP_NODELAY set
* Connected to kube-apiserver.mydomain.com (10.50.1.50) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: May 26 03:39:36 2019 GMT
*  expire date: May 25 03:39:36 2020 GMT
*  subjectAltName: host "kube-apiserver.mydomain.com" matched cert's "kube-apiserver.mydomain.com"
*  issuer: CN=kubernetes
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564287cbd900)
> GET / HTTP/2
> Host: kube-apiserver.mydomain.com:6443
> User-Agent: curl/7.58.0
> Accept: */*

我也可以在API服务器上使用curl查询api服务器： 

curl -v https://kube-apiserver.mydomain.com:6443
* Rebuilt URL to: https://kube-apiserver.mydomain.com:6443/
*   Trying 10.50.1.50...
* TCP_NODELAY set
* Connected to kube-apiserver.epc-instore.com (10.50.1.50) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: May 26 03:39:36 2019 GMT
*  expire date: May 25 03:39:36 2020 GMT
*  subjectAltName: host "kube-apiserver.mydomain.com" matched cert's "kube-apiserver.mydomain.com"
*  issuer: CN=kubernetes
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5628b9dbc900)
> GET / HTTP/2
> Host: kube-apiserver.mydomain.com:6443
> User-Agent: curl/7.58.0
> Accept: */*

api服务器上的清单包含： 

cat /etc/kubernetes/manifest/kube-apiserver.yaml
...
  - command:
    - kube-apiserver
    - --advertise-address=192.168.5.29
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-servers=http://etcd-cluster.mydomain.com:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.14.2
    imagePullPolicy: IfNotPresent
...

如果您对如何解决这个问题有任何想法或提示，我会全力以赴。 我对这个问题感到非常沮丧，现在它真的得到了我。 我将继续努力，但如果有人对这个问题有所了解，可以帮助它会很棒。

谢谢。

3 个解决方案

解决方案1
 0  2019-05-26 12:58:07

您当前的nginx配置未设置客户端证书。 ssl_certificate是服务器证书，如果您希望它向kubernetes-api-cluster提供客户端证书， kubernetes-api-cluster必须配置nginx以转发传入的客户端证书。 我以前使用proxy_set_header X-SSL-CERT $ssl_client_escaped_cert （ 文档 ）完成此操作 

upstream kubernetes-api-cluster { 
    server 192.168.5.19:6443;
    server 192.168.5.29:6443; 
} 

server { 
    listen 6443;
    ssl_certificate /etc/nginx/ssl/kube-apiserver.pem;
    ssl_certificate_key /etc/nginx/ssl/private/kube-apiserver.key;
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
    proxy_pass kubernetes-api-cluster; 

    #forward incoming client certificate
    ssl_verify_client optional; #requests the client certificate and verifies it if the certificate is present
    proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
}

解决方案2
 0  2019-05-27 18:10:49

这更像是一个故障排除的想法，真正针对问题的根源。 如果你能做到： 

kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes

从api服务器，你得到一个响应然后问题不是负载均衡器。 为了进一步证明这一点，您可以将相应的证书和文件复制到远程工作站并执行相同的操作： 

kubectl --kubeconfig [workstation location]/admin.conf get nodes

第二个显然暗示您可以直接访问负载均衡器。
如果这也有效，则可以确认证书是通过TCP负载均衡器传递的。

但是，当负载均衡器检查后端服务器的“可用性”时，错误将持续存在。 此检查不使用产生异常的证书。

解决方案3
 0 已采纳  

原始问题的实际根本原因是（引用此帖的作者@Daniel Maldonado）：

这是我的错误，我有一个防火墙配置错误，所有测试都表明负载平衡器正在探测kube-apiserver，而实际上并非如此。 这个问题完全是api-server本身的本地问题。 如果有人到达这一点，请确认API服务器可以从其自身获得所有端口，即环回。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 去TLS握手错误 来自…tls的TLS握手错误：客户端未提供证书 kubernetes“远程错误：tls：握手失败”缺少 SSL 证书 使用Go / TLS发送电子邮件会导致“远程错误：握手失败”响应 转到https客户端问题-远程错误：tls：握手失败 Kubernetes 活跃性和就绪性探测导致错误日志“http：来自 10.234.0.1:49330：EOF 的 TLS 握手错误” 具有TLS的保险柜-错误的证书错误 Docker 远程错误：tls：握手失败 TLS握手错误 从.NET到APN的SSL / TLS握手 - 远程证书无效
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM