Spinnaker on Kubernetes: Not able to start the spinnaker containers in kubernetes
Question
I was following this documentation to setup Spinnaker on Kubernetes. I ran the scripts as they specified. Then the replication controllers and services are started. But some of PODs are not started
root@nveeru~# kubectl get pods --namespace=spinnaker
NAME READY STATUS RESTARTS AGE
data-redis-master-v000-zsn7e 1/1 Running 0 2h
spin-clouddriver-v000-6yr88 1/1 Running 0 47m
spin-deck-v000-as4v7 1/1 Running 0 2h
spin-echo-v000-g737r 1/1 Running 0 2h
spin-front50-v000-v1g6e 0/1 CrashLoopBackOff 21 2h
spin-gate-v000-9k401 0/1 Running 0 2h
spin-igor-v000-zfc02 1/1 Running 0 2h
spin-orca-v000-umxj1 0/1 CrashLoopBackOff 20 2h
Then I kubectl describe the pods
root@veeru:~# kubectl describe pod spin-orca-v000-umxj1 --namespace=spinnaker
Name: spin-orca-v000-umxj1
Namespace: spinnaker
Node: 172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels: load-balancer-spin-orca=true,replication-controller=spin-orca-v000
Status: Running
IP: 172.16.33.8
Controllers: ReplicationController/spin-orca-v000
Containers:
orca:
Container ID: docker://e6d77e9fd92dc9614328d09a5bfda319dc7883b82f50cc352ff58dec2e933d04
Image: quay.io/spinnaker/orca:latest
Image ID: docker://sha256:2400633b89c1c7aa48e5195c040c669511238af9b55ff92201703895bd67a131
Port: 8083/TCP
QoS Tier:
cpu: BestEffort
memory: BestEffort
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 19 Sep 2016 02:59:09 -0700
Finished: Mon, 19 Sep 2016 02:59:39 -0700
Ready: False
Restart Count: 21
Readiness: http-get http://:8083/env delay=20s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
spinnaker-config:
Type: Secret (a volume populated by a Secret)
SecretName: spinnaker-config
default-token-6irrl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6irrl
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 3m 22 {kubelet 172.25.30.21} spec.containers{orca} Normal Pulling pulling image "quay.io/spinnaker/orca:latest"
1h 3m 22 {kubelet 172.25.30.21} spec.containers{orca} Normal Pulled Successfully pulled image "quay.io/spinnaker/orca:latest"
1h 3m 13 {kubelet 172.25.30.21} spec.containers{orca} Normal Created (events with common reason combined)
1h 3m 13 {kubelet 172.25.30.21} spec.containers{orca} Normal Started (events with common reason combined)
1h 3m 23 {kubelet 172.25.30.21} spec.containers{orca} Warning Unhealthy Readiness probe failed: Get http://172.16.33.8:8083/env: dial tcp 172.16.33.8:8083: connection refused
1h <invalid> 399 {kubelet 172.25.30.21} spec.containers{orca} Warning BackOff Back-off restarting failed docker container
1h <invalid> 373 {kubelet 172.25.30.21} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "orca" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=orca pod=spin-orca-v000-umxj1_spinnaker(ee2511f0-7e3d-11e6-ab16-0022195df673)"
spin-front50-v000-v1g6e
root@veeru:~# kubectl describe pod spin-front50-v000-v1g6e --namespace=spinnaker
Name: spin-front50-v000-v1g6e
Namespace: spinnaker
Node: 172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels: load-balancer-spin-front50=true,replication-controller=spin-front50-v000
Status: Running
IP: 172.16.33.9
Controllers: ReplicationController/spin-front50-v000
Containers:
front50:
Container ID: docker://f5559638e9ea4e30b3455ed9fea2ab1dd52be95f177b4b520a7e5bfbc033fc3b
Image: quay.io/spinnaker/front50:latest
Image ID: docker://sha256:e774808d76b096f45d85c43386c211a0a839c41c8d0dccb3b7ee62d17e977eb4
Port: 8080/TCP
QoS Tier:
memory: BestEffort
cpu: BestEffort
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 19 Sep 2016 03:02:08 -0700
Finished: Mon, 19 Sep 2016 03:02:15 -0700
Ready: False
Restart Count: 23
Readiness: http-get http://:8080/env delay=20s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
spinnaker-config:
Type: Secret (a volume populated by a Secret)
SecretName: spinnaker-config
creds-config:
Type: Secret (a volume populated by a Secret)
SecretName: creds-config
aws-config:
Type: Secret (a volume populated by a Secret)
SecretName: aws-config
default-token-6irrl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6irrl
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 3m 24 {kubelet 172.25.30.21} spec.containers{front50} Normal Pulling pulling image "quay.io/spinnaker/front50:latest"
1h 3m 24 {kubelet 172.25.30.21} spec.containers{front50} Normal Pulled Successfully pulled image "quay.io/spinnaker/front50:latest"
1h 3m 15 {kubelet 172.25.30.21} spec.containers{front50} Normal Created (events with common reason combined)
1h 3m 15 {kubelet 172.25.30.21} spec.containers{front50} Normal Started (events with common reason combined)
1h <invalid> 443 {kubelet 172.25.30.21} spec.containers{front50} Warning BackOff Back-off restarting failed docker container
1h <invalid> 417 {kubelet 172.25.30.21} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "front50" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=front50 pod=spin-front50-v000-v1g6e_spinnaker(edf85f41-7e3d-11e6-ab16-0022195df673)"
spin-gate-v000-9k401
root@n42-poweredge-5:~# kubectl describe pod spin-gate-v000-9k401 --namespace=spinnaker
Name: spin-gate-v000-9k401
Namespace: spinnaker
Node: 172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels: load-balancer-spin-gate=true,replication-controller=spin-gate-v000
Status: Running
IP: 172.16.33.6
Controllers: ReplicationController/spin-gate-v000
Containers:
gate:
Container ID: docker://7507c9d7c00e5834572cde2c0b0b54086288e9e30d3af161f0a1dbdf44672332
Image: quay.io/spinnaker/gate:latest
Image ID: docker://sha256:074d9616a43de8690c0a6a00345e422c903344f6876d9886f7357505082d06c7
Port: 8084/TCP
QoS Tier:
memory: BestEffort
cpu: BestEffort
State: Running
Started: Mon, 19 Sep 2016 01:14:54 -0700
Ready: False
Restart Count: 0
Readiness: http-get http://:8084/env delay=20s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
spinnaker-config:
Type: Secret (a volume populated by a Secret)
SecretName: spinnaker-config
default-token-6irrl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6irrl
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h <invalid> 696 {kubelet 172.25.30.21} spec.containers{gate} Warning Unhealthy Readiness probe failed: Get http://172.16.33.6:8084/env: dial tcp 172.16.33.6:8084: connection refused
what's wrong here?
UPDATE1
Logs (Please check the logs here )
2016-09-20 06:49:45.062 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:133)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:690)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134)
at org.springframework.boot.builder.SpringApplicationBuilder$run$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.netflix.spinnaker.front50.Main.main(Main.groovy:47)
Caused by: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.initialize(TomcatEmbeddedServletContainer.java:99)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.<init>(TomcatEmbeddedServletContainer.java:76)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getTomcatEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:384)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:156)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:159)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:130)
... 10 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration'........
..........
UPDATE-1(02-06-2017)
I tried above setup again in latest version of K8
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Still not all PODs are up
ubuntu@ip-172-31-18-78:~/spinnaker/experimental/kubernetes/simple$ kubectl get pods --namespace=spinnaker
NAME READY STATUS RESTARTS AGE
data-redis-master-v000-rzmzq 1/1 Running 0 31m
spin-clouddriver-v000-qhz97 1/1 Running 0 31m
spin-deck-v000-0sz8q 1/1 Running 0 31m
spin-echo-v000-q9xv5 1/1 Running 0 31m
spin-front50-v000-646vg 0/1 CrashLoopBackOff 10 31m
spin-gate-v000-vfvhg 0/1 Running 0 31m
spin-igor-v000-8j4r0 1/1 Running 0 31m
spin-orca-v000-ndpcx 0/1 CrashLoopBackOff 9 31m
Here is the logs links
Front50 https://pastebin.com/ge5TR4eR
Orca https://pastebin.com/wStmBtst
Gate https://pastebin.com/T8vjqL2K
Deck https://pastebin.com/kZnzN62W
Clouddriver https://pastebin.com/1pEU6V5D
Echo https://pastebin.com/cvJ4dVta
Igor https://pastebin.com/QYkHBxkr
Did I miss any configuration? I have not touched yaml config (Updated Jenkins URL,uname, passwd), that's I'm getting errors?. I'm new to Spinnaker. I had little knowledge on normal Spinnaker installation. Please guide me the installation.
Thanks
4 answers
solution1
1 2019-06-27 07:24:10
use halyard to install spinnaker. it is the recommended approach for deploying spinnaker in kubernetes clsuter
solution2
1 2019-07-19 19:03:35
Based on these errors, it seems that the Front50 service cannot reach any backend(if there is one configured). I would not modify the spinnaker-local.yml file directly, but instead use Halyard to install Spinnaker services into Kubernetes.
I have setup Spinnaker services in Kubernetes successfully using the instructions and scripts in this repo . Just omit/skip the components that you don't need.
solution3
1 2019-08-02 11:25:18
This question is very old. At the time I started working on Spinnaker, there was no Halyard or proper document on how to setup spinnaker on K8s. Since I see some attention getting on this question, I just want to provide some resource & tools to quickly setup.
solution4
0 2020-01-24 12:08:56
Might be the coredns issue. Orca mainly deals with establishing communication across the services in the namespace.
Try getting into some running pod (or use a busybox pod) to nslookup the spin-* services. Check the logs of coredns pods. Try restarting coredns pods (last option)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Spinnaker - Kubernetes - Can't find docker containers
Spinnaker with Kubernetes: Not able to connect the docker hub repository
Kubernetes on Spinnaker - Interservice communication
Error installing spinnaker on kubernetes
Spinnaker deployment in kubernetes is failing
How to Add Kubernetes Clusters to Spinnaker
Spinnaker unable to communicate with kubernetes cluster
Azure kubernetes - How to install spinnaker?
Is it safe/right to deploy Spinnaker to same kubernetes cluster which Spinnaker will manage?
Error when installing Spinnaker on Kubernetes on prem cluster
Related Tags