kublet doesn't start with cloud provider “openstack”

Question

To use cinder volumes I added the options --cloud-provider and --cloud-config to my kubelet configuration:

$ cat /etc/kubernetes/kubelet
###
# kubernetes kubelet (node) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname_override=192.168.100.76"

# location of the api-server
KUBELET_API_SERVER="--api_servers=https://localhost:6443"

# Add your own!
KUBELET_ARGS="--cluster_dns=10.100.0.10 --cluster_domain=cluster.local --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/manifests --cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud_config"

I've created a file with the necessary credentials:

$ sudo cat /etc/kubernetes/cloud_config
[Global]
auth-url=https://api.*********.com:5000/v2.0
user-id=kubecindertest
username=kubecindertest
password=*****
region=RegionOne
tenant-name=kubecindertest
tenant-id=6568768756a7886767e676f7efe76fe7
project-name=kubecindertest

When starting kubelet (manually), the process only logs unknown cloud provider "openstack" and exists:

source /etc/kubernetes/kubelet; sudo /usr/bin/kubelet $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBE_ALLOW_PRIV $KUBELET_ARGS
unknown cloud provider "openstack"

The openstack.go, defining the openstack provider in the kubernetes repository has the exact same name in lowercase:

const ProviderName = "openstack"

Update

It turned out the error was an expection that occured while parsing the config file. I removed all the optional or unwanted keys and now I use this as my config file:

$ sudo cat /etc/kubernetes/cloud_config
[Global]
auth-url=https://api.*********.com:5000/v2.0
username=kubecindertest
password=*****
region=RegionOne
tenant-id=6568768756a7886767e676f7efe76fe7

hower, starting the kublet only leads to another error:

I0923 07:14:33.315311   23743 manager.go:127] cAdvisor running in container: "/user.slice"
I0923 07:14:33.316263   23743 fs.go:93] Filesystem partitions: map[/dev/vda1:{mountpoint:/ major:253 minor:1}]
I0923 07:14:33.358848   23743 manager.go:158] Machine: {NumCores:2 CpuFrequency:2099998 MemoryCapacity:4144640000 MachineID:dae72fe0cc064eb0b7797f25bfaf69df SystemUUID:BEDAF943-624D-C04A-B92C-4EB07258246C BootID:e2d988e2-9aba-49bf-a344-fd62607a6754 Filesystems:[{Device:/dev/vda1 Capacity:21456445440}] DiskMap:map[252:0:{Name:dm-0 Major:252 Minor:0 Size:107374182400 Scheduler:none} 252:1:{Name:dm-1 Major:252 Minor:1 Size:10737418240 Scheduler:none} 252:2:{Name:dm-2 Major:252 Minor:2 Size:10737418240 Scheduler:none} 253:0:{Name:vda Major:253 Minor:0 Size:21474836480 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:fa:16:3e:64:fa:9a Speed:0 Mtu:1500} {Name:eth1 MacAddress:fa:16:3e:01:00:79 Speed:0 Mtu:1500} {Name:flannel.1 MacAddress:d2:9c:ad:29:df:c5 Speed:0 Mtu:1450}] Topology:[{Id:0 Memory:4294434816 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown}
I0923 07:14:33.363915   23743 manager.go:164] Version: {KernelVersion:3.10.0-229.11.1.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.8.1.el7 CadvisorVersion:0.16.0}
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x8559cd]

goroutine 1 [running]:
k8s.io/kubernetes/pkg/cloudprovider/providers/openstack.(*OpenStack).Instances(0x0, 0x0, 0x0, 0xe)
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/openstack/openstack.go:167 +0x8ed
k8s.io/kubernetes/cmd/kubelet/app.RunKubelet(0xc820144900, 0x0, 0x0, 0x0)
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:628 +0x13c
k8s.io/kubernetes/cmd/kubelet/app.(*KubeletServer).Run(0xc8202c2000, 0xc820144900, 0x0, 0x0)
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:420 +0x84b
main.main()
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:46 +0xab

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x1dc7000)
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x67
created by github.com/golang/glog.init.1
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x297

goroutine 37 [runnable]:
syscall.Syscall6(0x36, 0x4, 0x29, 0x1a, 0xc820035a7c, 0x4, 0x0, 0x0, 0x1a, 0x0)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:44 +0x5
syscall.setsockopt(0x4, 0x29, 0x1a, 0xc820035a7c, 0x4, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:1655 +0x73
syscall.SetsockoptInt(0x4, 0x29, 0x1a, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/syscall/syscall_unix.go:267 +0x61
net.setDefaultSockopts(0x4, 0xa, 0x1, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/sockopt_linux.go:17 +0x7f
net.socket(0x135f188, 0x3, 0xa, 0x1, 0x0, 0x0, 0x7fe031c3cd50, 0xc8205e43f0, 0x0, 0x0, ...)
    /usr/lib/golang/src/net/sock_posix.go:42 +0xcb
net.internetSocket(0x135f188, 0x3, 0x7fe031c3cd50, 0xc8205e43f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
    /usr/lib/golang/src/net/ipsock_posix.go:160 +0x141
net.ListenTCP(0x135f188, 0x3, 0xc8205e43f0, 0x7fe02fbf6bc0, 0x0, 0x0)
    /usr/lib/golang/src/net/tcpsock_posix.go:324 +0x19b
net.Listen(0x135f188, 0x3, 0xc8205ee100, 0x5, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/dial.go:393 +0x462
net/http.(*Server).ListenAndServe(0xc82007c660, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:1827 +0x8e
k8s.io/kubernetes/pkg/kubelet/cadvisor.(*cadvisorClient).exportHTTP.func1(0xc82007c660, 0x1062)
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:100 +0x7b
created by k8s.io/kubernetes/pkg/kubelet/cadvisor.(*cadvisorClient).exportHTTP
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:106 +0x3b6

goroutine 6 [syscall]:
os/signal.loop()
    /usr/lib/golang/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/lib/golang/src/os/signal/signal_unix.go:28 +0x37

goroutine 27 [sleep]:
time.Sleep(0x12a05f200)
    /usr/lib/golang/src/runtime/time.go:59 +0xf9
k8s.io/kubernetes/pkg/util.Until(0x164e780, 0x12a05f200, 0xc82007cc00)
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:117 +0x61
created by k8s.io/kubernetes/pkg/util.InitLogs
    /builddir/build/BUILD/kubernetes-196f58b9cb25a2222c7f9aacd624737910b03acb/_output/local/go/src/k8s.io/kubernetes/pkg/util/logs.go:49 +0xba

Update 2

The second problem was a certificate issue.

Answer 1

The error messages are not so clear in the openstack cloud provider plugin. It wont register if the there is any unexpected field in the config file, it will throw an error, which unfortunately is not properly bubbled.

Here are the fields (and their identifiers) that are recognized by the plugin

[Global] AuthUrl string gcfg:"auth-url" Username string UserId string gcfg:"user-id" Password string ApiKey string gcfg:"api-key" TenantId string gcfg:"tenant-id" TenantName string gcfg:"tenant-name" DomainId string gcfg:"domain-id" DomainName string gcfg:"domain-name" Region string

Offcourse not all of them are required. I usually have

[Global] auth-url username password region tenant-id