Jenkins超时等待容器准备就绪

Question

I'm running Jenkins on OpenShift Origin and Fabric8.

OpenShift Master: v1.3.2 Kubernetes Master: v1.3.0+52492b4

I'm having boat loads of problems with jenkins and farbic8!

One of the latest issues is an error - happens everytime - when trying to build in Jenkins.

Anyone know how to fix this?

The error is:

Waiting for container container [maven] of pod [jenkins-1-1d0ts] to become ready.
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
java.io.IOException: Failed to execute shell script inside container [maven] of pod [jenkins-1-1d0ts]. Timed out waiting for container to become ready!
    at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:88)
    at hudson.Launcher$ProcStarter.start(Launcher.java:384)
    at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:147)
    at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:172)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
    at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:103)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
    at mavenCanaryRelease.call(/var/jenkins_home/jobs/shiftwork-staffservice-pipeline-dev/builds/6/libs/github.com/fabric8io/fabric8-pipeline-library/vars/mavenCanaryRelease.groovy:11)
    at WorkflowScript.run(WorkflowScript:39)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:58)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

I also noticed the following. There is nothing unusual going on with the pods. But it seems Jenkins cannot connect to Kubenetes.

All pods appear to be running correctly.

When I retrieve the logs from the Jenkins pod I see:

INFO: Finished Download metadata. 6,467 ms
--> setting agent port for jnlp
--> setting agent port for jnlp... done
Jan 23, 2017 9:11:08 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Gravatar periodic lookup
Jan 23, 2017 9:11:08 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Gravatar periodic lookup. 1 ms
Jan 23, 2017 9:11:23 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: shiftwork-staffservice-pipeline-dev #21 completed: FAILURE
Jan 23, 2017 9:11:23 PM io.jenkins.blueocean.events.PipelineEventListener$1 run
SEVERE: Unexpected error publishing pipeline FlowNode event.
java.util.concurrent.ExecutionException: hudson.AbortException
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
    at io.jenkins.blueocean.events.PipelineEventListener$1.run(PipelineEventListener.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.AbortException
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1068)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1094)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:57)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:215)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:404)

Answer 1

This looks like the same Jenkins issue I reported here https://issues.jenkins-ci.org/browse/JENKINS-41388

TL;DR If Jenkins is running in a namespace other than 'default' then any change to jenkins configuration incorrectly also updates the namespace to run build pods. Until this is fixed in Jenkins a workaround is to manually change that 'Kubernetes Namespace' field to the same namespace as Jenkins is running. This should avoid the issue with the service account.