Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object

Question

Trying to use Cinder volumens on OpenStack as persistent volumes for my pods. As soon as I configure the cloudprovider and restart the kubelet, the kubelet fails to get its external ID from the cloud provider.

The OpenStack API is reachable via https using a comodo certificate. the comodo-ca-bundle is installed as trusted ca on the node. Using curl against the API works without --insecure and --cacert options.

Using kubernetes 1.1.0-alpha on centos 7

$ sudo journalctl -u kubelet

Oct 01 07:40:26 [4196]: I1001 07:40:26.303887    4196 debugging.go:129]     Content-Length: 1159
Oct 01 07:40:26 [4196]: I1001 07:40:26.303895    4196 debugging.go:129]     Content-Type: application/json
Oct 01 07:40:26 [4196]: I1001 07:40:26.303950    4196 request.go:755] Response Body: {"kind":"NodeList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/nodes","resourceVersion":"172921"},"items":[{"metadata":{"name":"192.168.100.80","selfLink":"/api/v1/nodes/192.168.100.80","uid":"b48b4cb9-676f-11e5-8521-fa163ef34ff1","resourceVersion":"172900","creationTimestamp":"2015-09-30T12:35:17Z","labels":{"kubernetes.io/hostname":"192.168.100.80"}},"spec":{"externalID":"192.168.100.80"},"status":{"capacity":{"cpu":"2","memory":"4047500Ki","pods":"40"},"conditions":[{"type":"Ready","status":"Unknown","lastHeartbeatTime":"2015-10-01T07:31:55Z","lastTransitionTime":"2015-10-01T07:32:36Z","reason":"Kubelet stopped posting node status."}],"addresses":[{"type":"LegacyHostIP","address":"192.168.100.80"},{"type":"InternalIP","address":"192.168.100.80"}],"nodeInfo":{"machineID":"dae72fe0cc064eb0b7797f25bfaf69df","systemUUID":"384A8E40-1296-9A42-AD77-445D83BB5888","bootID":"5c7eb3ff-d86f-41f2-b3eb-a39adf313a4f","kernelVersion":"3.10.0-229.14.1.el7.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://1.7.1","kubeletVersion":"v1.1.0-alpha.1.390+196f58b9cb25a2","kubeProxyVersion":"v1.1.0-alpha.1.390+196f58b9cb25a2"}}}]}
Oct 01 07:40:26 [4196]: I1001 07:40:26.475016    4196 request.go:457] Request Body: {"kind":"DeleteOptions","apiVersion":"v1","gracePeriodSeconds":0}
Oct 01 07:40:26 [4196]: I1001 07:40:26.475148    4196 debugging.go:101] curl -k -v -XDELETE  -H "Authorization: Bearer rhARkbozkWcrJyvdLQqF9TNO86KHjOsq" -H "User-Agent: kubelet/v1.1.0 (linux/amd64) kubernetes/196f58b" https://localhost:6443/api/v1/namespaces/kube-system/pods/fluentd-elasticsearch-192.168.100.80
Oct 01 07:40:26 [4196]: I1001 07:40:26.526794    4196 debugging.go:120] DELETE https://localhost:6443/api/v1/namespaces/kube-system/pods/fluentd-elasticsearch-192.168.100.80 200 OK in 51 milliseconds
Oct 01 07:40:26 [4196]: I1001 07:40:26.526865    4196 debugging.go:126] Response Headers:
Oct 01 07:40:26 [4196]: I1001 07:40:26.526897    4196 debugging.go:129]     Content-Type: application/json
Oct 01 07:40:26 [4196]: I1001 07:40:26.526927    4196 debugging.go:129]     Date: Thu, 01 Oct 2015 07:40:26 GMT
Oct 01 07:40:26 [4196]: I1001 07:40:26.526957    4196 debugging.go:129]     Content-Length: 1977
Oct 01 07:40:26 [4196]: I1001 07:40:26.527056    4196 request.go:755] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"fluentd-elasticsearch-192.168.100.80","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/pods/fluentd-elasticsearch-192.168.100.80","uid":"a90941f6-680f-11e5-988c-fa163e94cde4","resourceVersion":"172926","creationTimestamp":"2015-10-01T07:40:17Z","deletionTimestamp":"2015-10-01T07:40:26Z","deletionGracePeriodSeconds":0,"annotations":{"kubernetes.io/config.mirror":"mirror","kubernetes.io/config.seen":"2015-10-01T07:39:43.986114806Z","kubernetes.io/config.source":"file"}},"spec":{"volumes":[{"name":"varlog","hostPath":{"path":"/var/log"}},{"name":"varlibdockercontainers","hostPath":{"path":"/var/lib/docker/containers"}}],"containers":[{"name":"fluentd-elasticsearch","image":"gcr.io/google_containers/fluentd-elasticsearch:1.11","args":["-q"],"resources":{"limits":{"cpu":"100m"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"varlog","mountPath":"/var/log"},{"name":"varlibdockercontainers","readOnly":true,"mountPath":"/var/lib/docker/containers"}],"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeName":"192.168.100.80"},"status":{"phase":"Running","conditions":[{"type":"Ready","status":"True"}],"hostIP":"192.168.100.80","podIP":"172.16.58.24","startTime":"2015-10-01T07:40:17Z","containerStatuses":[{"name":"fluentd-elasticsearch","state":{"running":{"startedAt":"2015-10-01T07:37:23Z"}},"lastState":{"terminated":{"exitCode":137,"startedAt":"2015-10-01T07:23:00Z","finishedAt":"2015-10-01T07:33:17Z","containerID":"docker://1398736fd9b274132721206ccaf89030af5e8e304118d29286aec6b2529395ee"}},"ready":true,"restartCount":1,"image":"gcr.io/google_containers/fluentd-elasticsearch:1.11","imageID":"docker://03ba3d224c2a80600a0b44a9894ac0de5526d36b810b13924e33ada76f1e7406","containerID":"docker://d9ac24c8a0fbceea7c494bce73d56d6ea5f003f1d1b7b8ad3975fc7e3c7679b4"}]}}
Oct 01 07:40:26 [4196]: I1001 07:40:26.528210    4196 status_manager.go:209] Pod "fluentd-elasticsearch-192.168.100.80" fully terminated and removed from etcd
Oct 01 07:40:26 [4196]: I1001 07:40:26.675178    4196 debugging.go:101] curl -k -v -XGET  -H "User-Agent: kubelet/v1.1.0 (linux/amd64) kubernetes/196f58b" -H "Authorization: Bearer rhARkbozkWcrJyvdLQqF9TNO86KHjOsq" https://localhost:6443/api/v1/services
Oct 01 07:40:26 [4196]: I1001 07:40:26.710214    4196 debugging.go:120] GET https://localhost:6443/api/v1/services 200 OK in 34 milliseconds
Oct 01 07:40:26 [4196]: I1001 07:40:26.710249    4196 debugging.go:126] Response Headers:
Oct 01 07:40:26 [4196]: I1001 07:40:26.710260    4196 debugging.go:129]     Content-Type: application/json
Oct 01 07:40:26 [4196]: I1001 07:40:26.710270    4196 debugging.go:129]     Date: Thu, 01 Oct 2015 07:40:26 GMT
Oct 01 07:40:26 [4196]: I1001 07:40:26.710436    4196 request.go:755] Response Body: {"kind":"ServiceList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/services","resourceVersion":"172927"},"items":[{"metadata":{"name":"kubernetes","namespace":"default","selfLink":"/api/v1/namespaces/default/services/kubernetes","uid":"28717019-676b-11e5-afb9-fa163e94cde4","resourceVersion":"18","creationTimestamp":"2015-09-30T12:02:44Z","labels":{"component":"apiserver","provider":"kubernetes"}},"spec":{"ports":[{"protocol":"TCP","port":443,"targetPort":443}],"clusterIP":"10.100.0.1","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"elasticsearch-logging","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/services/elasticsearch-logging","uid":"833c8df5-676b-11e5-958e-fa163e94cde4","resourceVersion":"153","creationTimestamp":"2015-09-30T12:05:16Z","labels":{"k8s-app":"elasticsearch-logging","kubernetes.io/cluster-service":"true","kubernetes.io/name":"Elasticsearch"}},"spec":{"ports":[{"protocol":"TCP","port":9200,"targetPort":"db"}],"selector":{"k8s-app":"elasticsearch-logging"},"clusterIP":"10.100.3.159","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"kibana-logging","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/services/kibana-logging","uid":"833043fa-676b-11e5-958e-fa163e94cde4","resourceVersion":"149","creationTimestamp":"2015-09-30T12:05:16Z","labels":{"k8s-app":"kibana-logging","kubernetes.io/cluster-service":"true","kubernetes.io/name":"Kibana"}},"spec":{"ports":[{"protocol":"TCP","port":5601,"targetPort":"ui"}],"selector":{"k8s-app":"kibana-logging"},"clusterIP":"10.100.136.111","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"kube-dns","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/services/kube-dns","uid":"8319ba13-676b-11e5-958e-fa163e94cde4","resourceVersion":"146","creationTimestamp":"2015-09-30T12:05:16Z","labels":{"k8s-app":"kube-dns
Oct 01 07:40:26 [4196]: ","kubernetes.io/cluster-service":"true","kubernetes.io/name":"KubeDNS"}},"spec":{"ports":[{"name":"dns","protocol":"UDP","port":53,"targetPort":53},{"name":"dns-tcp","protocol":"TCP","port":53,"targetPort":53}],"selector":{"k8s-app":"kube-dns"},"clusterIP":"10.100.0.10","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"kube-ui","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/services/kube-ui","uid":"83473271-676b-11e5-958e-fa163e94cde4","resourceVersion":"155","creationTimestamp":"2015-09-30T12:05:16Z","labels":{"k8s-app":"kube-ui","kubernetes.io/cluster-service":"true","kubernetes.io/name":"KubeUI"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":8080}],"selector":{"k8s-app":"kube-ui"},"clusterIP":"10.100.246.61","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"monitoring-grafana","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/services/monitoring-grafana","uid":"835da09c-676b-11e5-958e-fa163e94cde4","resourceVersion":"157","creationTimestamp":"2015-09-30T12:05:16Z","labels":{"kubernetes.io/cluster-service":"true","kubernetes.io/name":"Grafana"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":8080}],"selector":{"k8s-app":"influxGrafana"},"clusterIP":"10.100.207.92","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"monitoring-heapster","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/services/monitoring-heapster","uid":"83367b90-676b-11e5-958e-fa163e94cde4","resourceVersion":"151","creationTimestamp":"2015-09-30T12:05:16Z","labels":{"kubernetes.io/cluster-service":"true","kubernetes.io/name":"Heapster"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":8082}],"selector":{"k8s-app":"heapster"},"clusterIP":"10.100.119.4","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"monitoring-influxdb","namespace":"kube-system","selfLink":"/api/v1/names
Oct 01 07:40:26 [4196]: paces/kube-system/services/monitoring-influxdb","uid":"836c95b8-676b-11e5-958e-fa163e94cde4","resourceVersion":"159","creationTimestamp":"2015-09-30T12:05:16Z","labels":{"kubernetes.io/cluster-service":"true","kubernetes.io/name":"InfluxDB"}},"spec":{"ports":[{"name":"http","protocol":"TCP","port":8083,"targetPort":8083},{"name":"api","protocol":"TCP","port":8086,"targetPort":8086}],"selector":{"k8s-app":"influxGrafana"},"clusterIP":"10.100.101.182","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},{"metadata":{"name":"reverseproxy","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/services/reverseproxy","uid":"15e65b7d-6776-11e5-a5d0-fa163e94cde4","resourceVersion":"10994","creationTimestamp":"2015-09-30T13:20:57Z","labels":{"k8s-app":"reverseproxy","kubernetes.io/cluster-service":"true","kubernetes.io/name":"reverseproxy"}},"spec":{"ports":[{"name":"http","protocol":"TCP","port":8181,"targetPort":8181,"nodePort":80},{"name":"https","protocol":"TCP","port":8181,"targetPort":8181,"nodePort":443}],"selector":{"k8s-app":"reverseproxy"},"clusterIP":"10.100.168.84","type":"NodePort","sessionAffinity":"None"},"status":{"loadBalancer":{}}}]}
Oct 01 07:40:26 [4196]: I1001 07:40:26.875150    4196 debugging.go:101] curl -k -v -XGET  -H "User-Agent: kubelet/v1.1.0 (linux/amd64) kubernetes/196f58b" -H "Authorization: Bearer rhARkbozkWcrJyvdLQqF9TNO86KHjOsq" https://localhost:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3D192.168.100.80&resourceVersion=172921
Oct 01 07:40:26 [4196]: I1001 07:40:26.900981    4196 debugging.go:120] GET https://localhost:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3D192.168.100.80&resourceVersion=172921 200 OK in 25 milliseconds
Oct 01 07:40:26 [4196]: I1001 07:40:26.901009    4196 debugging.go:126] Response Headers:
Oct 01 07:40:26 [4196]: I1001 07:40:26.901018    4196 debugging.go:129]     Date: Thu, 01 Oct 2015 07:40:26 GMT
Oct 01 07:40:27 [4196]: I1001 07:40:27.001744    4196 iowatcher.go:102] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 01 07:40:27 [4196]: I1001 07:40:27.002685    4196 reflector.go:294] pkg/client/unversioned/cache/reflector.go:87: Unexpected watch close - watch lasted less than a second and no items received
Oct 01 07:40:27 [4196]: W1001 07:40:27.002716    4196 reflector.go:224] pkg/client/unversioned/cache/reflector.go:87: watch of *api.Node ended with: very short watch
Oct 01 07:40:27 [4196]: I1001 07:40:27.075065    4196 debugging.go:101] curl -k -v -XGET  -H "User-Agent: kubelet/v1.1.0 (linux/amd64) kubernetes/196f58b" -H "Authorization: Bearer rhARkbozkWcrJyvdLQqF9TNO86KHjOsq" https://localhost:6443/api/v1/watch/services?resourceVersion=172927
Oct 01 07:40:27 [4196]: I1001 07:40:27.101642    4196 debugging.go:120] GET https://localhost:6443/api/v1/watch/services?resourceVersion=172927 200 OK in 26 milliseconds
Oct 01 07:40:27 [4196]: I1001 07:40:27.101689    4196 debugging.go:126] Response Headers:
Oct 01 07:40:27 [4196]: I1001 07:40:27.101705    4196 debugging.go:129]     Date: Thu, 01 Oct 2015 07:40:27 GMT
Oct 01 07:40:27 [4196]: I1001 07:40:27.104168    4196 openstack.go:164] openstack.Instances() called
Oct 01 07:40:27 [4196]: I1001 07:40:27.133478    4196 openstack.go:201] Found 8 compute flavors
Oct 01 07:40:27 [4196]: I1001 07:40:27.133519    4196 openstack.go:202] Claiming to support Instances
Oct 01 07:40:27 [4196]: E1001 07:40:27.158908    4196 kubelet.go:846] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object
Oct 01 07:40:27 [4196]: I1001 07:40:27.202978    4196 iowatcher.go:102] Unexpected EOF during watch stream event decoding: unexpected EOF
Oct 01 07:40:27 [4196]: I1001 07:40:27.203110    4196 reflector.go:294] pkg/client/unversioned/cache/reflector.go:87: Unexpected watch close - watch lasted less than a second and no items received
Oct 01 07:40:27 [4196]: W1001 07:40:27.203136    4196 reflector.go:224] pkg/client/unversioned/cache/reflector.go:87: watch of *api.Service ended with: very short watch
Oct 01 07:40:27 [4196]: I1001 07:40:27.275208    4196 debugging.go:101] curl -k -v -XGET  -H "Authorization: Bearer rhARkbozkWcrJyvdLQqF9TNO86KHjOsq" -H "User-Agent: kubelet/v1.1.0 (linux/amd64) kubernetes/196f58b" https://localhost:6443/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.100.80
Oct 01 07:40:27 [4196]: I1001 07:40:27.308434    4196 debugging.go:120] GET https://localhost:6443/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.100.80 200 OK in 33 milliseconds
Oct 01 07:40:27 [4196]: I1001 07:40:27.308464    4196 debugging.go:126] Response Headers:
Oct 01 07:40:27 [4196]: I1001 07:40:27.308475    4196 debugging.go:129]     Content-Type: application/json
Oct 01 07:40:27 [4196]: I1001 07:40:27.308484    4196 debugging.go:129]     Date: Thu, 01 Oct 2015 07:40:27 GMT
Oct 01 07:40:27 [4196]: I1001 07:40:27.308491    4196 debugging.go:129]     Content-Length: 113
Oct 01 07:40:27 [4196]: I1001 07:40:27.308524    4196 request.go:755] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/pods","resourceVersion":"172941"},"items":[]}
Oct 01 07:40:27 [4196]: I1001 07:40:27.308719    4196 config.go:252] Setting pods for source api
Oct 01 07:40:27 [4196]: I1001 07:40:27.308753    4196 kubelet.go:1921] SyncLoop (REMOVE): "fluentd-elasticsearch-192.168.100.80_kube-system"
Oct 01 07:40:27 [4196]: I1001 07:40:27.308931    4196 volumes.go:100] Used volume plugin "kubernetes.io/host-path" for varlog
Oct 01 07:40:27 [4196]: I1001 07:40:27.308960    4196 volumes.go:100] Used volume plugin "kubernetes.io/host-path" for varlibdockercontainers
Oct 01 07:40:27 [4196]: I1001 07:40:27.308977    4196 kubelet.go:2531] Generating status for "fluentd-elasticsearch-192.168.100.80_kube-system"

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"1+",
GitVersion:"v1.1.0-alpha.1.390+196f58b9cb25a2", 
GitCommit:"196f58b9cb25a2222c7f9aacd624737910b03acb",
GitTreeState:"clean"}

Server Version: version.Info{Major:"1", Minor:"1+",
GitVersion:"v1.1.0-alpha.1.390+196f58b9cb25a2",
GitCommit:
"196f58b9cb25a2222c7f9aacd624737910b03acb",
GitTreeState:"clean"}

$ cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ cat /etc/kubernetes/kubelet

###
# kubernetes kubelet (node) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname_override=192.168.100.80"

# location of the api-server
KUBELET_API_SERVER="--api_servers=https://localhost:6443"

# Add your own!
KUBELET_ARGS="--cluster_dns=10.100.0.10 --cluster_domain=cluster.local --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/manifests --v=9 --cloud-config=/etc/kubernetes/cloud_config  --cloud-provider=openstack --machine-id-file=/etc/machine-id"

$ cat /etc/kubernetes/cloud_config

[Global]
auth-url=https://api.*******.de:5000/v2.0
username=username
password=password
region=RegionOne
tenant-id=4ee7b21351d94f2b96d363efe131b833

Answer 1

Kubelet is able to reach Openstack, however it is failing to find this node in the list of servers, in this tenant, and in this region. Oct 01 07:40:27 [4196]: I1001 07:40:27.133478 4196 openstack.go:201] Found 8 compute flavors Oct 01 07:40:27 [4196]: E1001 07:40:27.158908 4196 kubelet.go:846] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object

Node's hostname is used to identify it from the list of servers provided by cloud provider. However, it can be overridden using --hostname_overide flag.

In your config, I see that you have overridden it with an ip, if this does not match the name of the server as reported by Nova, you are likely to get this error.