Kubernetes 上的 Spinnaker：无法在 Kubernetes 中启动 Spinnaker 容器

Question

I was following this documentation to setup Spinnaker on Kubernetes. I ran the scripts as they specified. Then the replication controllers and services are started. But some of PODs are not started

root@nveeru~# kubectl get pods --namespace=spinnaker
NAME                           READY     STATUS             RESTARTS   AGE
data-redis-master-v000-zsn7e   1/1       Running            0          2h
spin-clouddriver-v000-6yr88    1/1       Running            0          47m
spin-deck-v000-as4v7           1/1       Running            0          2h
spin-echo-v000-g737r           1/1       Running            0          2h
spin-front50-v000-v1g6e        0/1       CrashLoopBackOff   21         2h
spin-gate-v000-9k401           0/1       Running            0          2h
spin-igor-v000-zfc02           1/1       Running            0          2h
spin-orca-v000-umxj1           0/1       CrashLoopBackOff   20         2h

Then I kubectl describe the pods

root@veeru:~# kubectl describe pod spin-orca-v000-umxj1 --namespace=spinnaker
Name:       spin-orca-v000-umxj1
Namespace:  spinnaker
Node:       172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels:     load-balancer-spin-orca=true,replication-controller=spin-orca-v000
Status:     Running
IP:     172.16.33.8
Controllers:    ReplicationController/spin-orca-v000
Containers:
  orca:
    Container ID:   docker://e6d77e9fd92dc9614328d09a5bfda319dc7883b82f50cc352ff58dec2e933d04
    Image:      quay.io/spinnaker/orca:latest
    Image ID:       docker://sha256:2400633b89c1c7aa48e5195c040c669511238af9b55ff92201703895bd67a131
    Port:       8083/TCP
    QoS Tier:
      cpu:      BestEffort
      memory:       BestEffort
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 19 Sep 2016 02:59:09 -0700
      Finished:     Mon, 19 Sep 2016 02:59:39 -0700
    Ready:      False
    Restart Count:  21
    Readiness:      http-get http://:8083/env delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  spinnaker-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: spinnaker-config
  default-token-6irrl:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-6irrl
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath       Type        Reason      Message
  --------- --------    -----   ----            -------------       --------    ------      -------
  1h        3m      22  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Pulling     pulling image "quay.io/spinnaker/orca:latest"
  1h        3m      22  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Pulled      Successfully pulled image "quay.io/spinnaker/orca:latest"
  1h        3m      13  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Created     (events with common reason combined)
  1h        3m      13  {kubelet 172.25.30.21}  spec.containers{orca}   Normal      Started     (events with common reason combined)
  1h        3m      23  {kubelet 172.25.30.21}  spec.containers{orca}   Warning     Unhealthy   Readiness probe failed: Get http://172.16.33.8:8083/env: dial tcp 172.16.33.8:8083: connection refused
  1h        <invalid>   399 {kubelet 172.25.30.21}  spec.containers{orca}   Warning     BackOff     Back-off restarting failed docker container
  1h        <invalid>   373 {kubelet 172.25.30.21}              Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "orca" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=orca pod=spin-orca-v000-umxj1_spinnaker(ee2511f0-7e3d-11e6-ab16-0022195df673)"

spin-front50-v000-v1g6e

root@veeru:~# kubectl describe pod spin-front50-v000-v1g6e  --namespace=spinnaker
Name:       spin-front50-v000-v1g6e
Namespace:  spinnaker
Node:       172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels:     load-balancer-spin-front50=true,replication-controller=spin-front50-v000
Status:     Running
IP:     172.16.33.9
Controllers:    ReplicationController/spin-front50-v000
Containers:
  front50:
    Container ID:   docker://f5559638e9ea4e30b3455ed9fea2ab1dd52be95f177b4b520a7e5bfbc033fc3b
    Image:      quay.io/spinnaker/front50:latest
    Image ID:       docker://sha256:e774808d76b096f45d85c43386c211a0a839c41c8d0dccb3b7ee62d17e977eb4
    Port:       8080/TCP
    QoS Tier:
      memory:       BestEffort
      cpu:      BestEffort
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 19 Sep 2016 03:02:08 -0700
      Finished:     Mon, 19 Sep 2016 03:02:15 -0700
    Ready:      False
    Restart Count:  23
    Readiness:      http-get http://:8080/env delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  spinnaker-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: spinnaker-config
  creds-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: creds-config
  aws-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: aws-config
  default-token-6irrl:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-6irrl
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath           Type        Reason      Message
  --------- --------    -----   ----            -------------           --------    ------      -------
  1h        3m      24  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Pulling     pulling image "quay.io/spinnaker/front50:latest"
  1h        3m      24  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Pulled      Successfully pulled image "quay.io/spinnaker/front50:latest"
  1h        3m      15  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Created     (events with common reason combined)
  1h        3m      15  {kubelet 172.25.30.21}  spec.containers{front50}    Normal      Started     (events with common reason combined)
  1h        <invalid>   443 {kubelet 172.25.30.21}  spec.containers{front50}    Warning     BackOff     Back-off restarting failed docker container
  1h        <invalid>   417 {kubelet 172.25.30.21}                  Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "front50" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=front50 pod=spin-front50-v000-v1g6e_spinnaker(edf85f41-7e3d-11e6-ab16-0022195df673)"

spin-gate-v000-9k401

root@n42-poweredge-5:~# kubectl describe pod spin-gate-v000-9k401 --namespace=spinnaker
Name:       spin-gate-v000-9k401
Namespace:  spinnaker
Node:       172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels:     load-balancer-spin-gate=true,replication-controller=spin-gate-v000
Status:     Running
IP:     172.16.33.6
Controllers:    ReplicationController/spin-gate-v000
Containers:
  gate:
    Container ID:   docker://7507c9d7c00e5834572cde2c0b0b54086288e9e30d3af161f0a1dbdf44672332
    Image:      quay.io/spinnaker/gate:latest
    Image ID:       docker://sha256:074d9616a43de8690c0a6a00345e422c903344f6876d9886f7357505082d06c7
    Port:       8084/TCP
    QoS Tier:
      memory:       BestEffort
      cpu:      BestEffort
    State:      Running
      Started:      Mon, 19 Sep 2016 01:14:54 -0700
    Ready:      False
    Restart Count:  0
    Readiness:      http-get http://:8084/env delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment Variables:
Conditions:
  Type      Status
  Ready     False 
Volumes:
  spinnaker-config:
    Type:   Secret (a volume populated by a Secret)
    SecretName: spinnaker-config
  default-token-6irrl:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-6irrl
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath       Type        Reason      Message
  --------- --------    -----   ----            -------------       --------    ------      -------
  1h        <invalid>   696 {kubelet 172.25.30.21}  spec.containers{gate}   Warning     Unhealthy   Readiness probe failed: Get http://172.16.33.6:8084/env: dial tcp 172.16.33.6:8084: connection refused

what's wrong here?

UPDATE1

Logs (Please check the logs here )

2016-09-20 06:49:45.062 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application startup failed

org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:133)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:690)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134)
    at org.springframework.boot.builder.SpringApplicationBuilder$run$0.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at com.netflix.spinnaker.front50.Main.main(Main.groovy:47)
Caused by: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.initialize(TomcatEmbeddedServletContainer.java:99)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.<init>(TomcatEmbeddedServletContainer.java:76)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getTomcatEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:384)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:156)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:159)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:130)
    ... 10 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration'........
..........

UPDATE-1(02-06-2017)

I tried above setup again in latest version of K8

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Still not all PODs are up

ubuntu@ip-172-31-18-78:~/spinnaker/experimental/kubernetes/simple$ kubectl get pods --namespace=spinnaker
NAME                           READY     STATUS             RESTARTS   AGE
data-redis-master-v000-rzmzq   1/1       Running            0          31m
spin-clouddriver-v000-qhz97    1/1       Running            0          31m
spin-deck-v000-0sz8q           1/1       Running            0          31m
spin-echo-v000-q9xv5           1/1       Running            0          31m
spin-front50-v000-646vg        0/1       CrashLoopBackOff   10         31m
spin-gate-v000-vfvhg           0/1       Running            0          31m
spin-igor-v000-8j4r0           1/1       Running            0          31m
spin-orca-v000-ndpcx           0/1       CrashLoopBackOff   9          31m

Here is the logs links

Front50 https://pastebin.com/ge5TR4eR

Orca https://pastebin.com/wStmBtst

Gate https://pastebin.com/T8vjqL2K

Deck https://pastebin.com/kZnzN62W

Clouddriver https://pastebin.com/1pEU6V5D

Echo https://pastebin.com/cvJ4dVta

Igor https://pastebin.com/QYkHBxkr

Did I miss any configuration? I have not touched yaml config (Updated Jenkins URL,uname, passwd), that's I'm getting errors?. I'm new to Spinnaker. I had little knowledge on normal Spinnaker installation. Please guide me the installation.

Thanks

Answer 1

use halyard to install spinnaker. it is the recommended approach for deploying spinnaker in kubernetes clsuter

Answer 2

Based on these errors, it seems that the Front50 service cannot reach any backend(if there is one configured). I would not modify the spinnaker-local.yml file directly, but instead use Halyard to install Spinnaker services into Kubernetes.

I have setup Spinnaker services in Kubernetes successfully using the instructions and scripts in this repo . Just omit/skip the components that you don't need.

https://github.com/grizzthedj/kubernetes-spinnaker

Answer 3

This question is very old. At the time I started working on Spinnaker, there was no Halyard or proper document on how to setup spinnaker on K8s. Since I see some attention getting on this question, I just want to provide some resource & tools to quickly setup.

• Spinnaker Installation
• Guides; How to use spinnaker with some example
• Some useful scripts & K8s deployment files that I created

Answer 4

Might be the coredns issue. Orca mainly deals with establishing communication across the services in the namespace.

Try getting into some running pod (or use a busybox pod) to nslookup the spin-* services. Check the logs of coredns pods. Try restarting coredns pods (last option)