Intermittent Build Failure on Jenkins

Question

I have project in Jenkins (running on linux) that polls SCM at defined interval and builds on new changes. I am getting build failures intermittently with the following Console Output:

Started by user MyUser User
Building on master in workspace <http://ci.mybuild.com:8080/job/temp-project/ws/>
Checkout:workspace / <http://ci.mybuild.com:8080/job/temp-project/ws/> - hudson.remoting.LocalChannel@5f2c402a
Using strategy: Default
Cloning the remote Git repository
Cloning repository origin
FATAL: hudson/FilePath$11
java.lang.NoClassDefFoundError: hudson/FilePath$11
    at hudson.FilePath.deleteRecursive(FilePath.java:980)
    at hudson.plugins.git.GitAPI.clone(GitAPI.java:211)
    at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1121)
    at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1063)
    at hudson.FilePath.act(FilePath.java:851)
    at hudson.FilePath.act(FilePath.java:824)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1063)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1308)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
    at hudson.model.Run.execute(Run.java:1516)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:236)

I tried creating a new job with similar configurations, but the first build failed with this error.

Can someone please help me with some pointers on this?

Answer 1

See in the log - deleteRecursive is failing:

...
java.lang.NoClassDefFoundError: hudson/FilePath$11
    at hudson.FilePath.deleteRecursive(FilePath.java:980)
    at hudson.plugins.git.GitAPI.clone(GitAPI.java:211)
...

It probably means your file-system prevents deleting of files in that area -
please check file-permissions there (also on folders).

Answer 2

This is almost certainly happening because Jenkins is trying to clean up/delete a file which is concurrently being accessed by another process.

Try running the following command right after you see this error:

lsof /my/repo/dir

This will tell you which files are locked in your repo directory and what is locking them.

Answer 3

The problem

The nature of the exception is one of classloading. Based on the actual exception:

java.lang.NoClassDefFoundError: hudson/FilePath$11

Means that the JVM cannot find the class definition (obviously). The dollar sign and number ($11) after "FilePath" indicates that it cannot find an anonymous inner class defined within the FilePath class, which can be seen at:

https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/FilePath.java

This supports this theory, since there are anonymous implementations of "FileCallable".

The solution

Now for the hard part, what to do about it.

I suspect that you are running jenkins in some form of web container (not known in the present form of the question). Web containers are notorious for their classloading idiosyncrasies. If you can, try running jenkins standalone, or in another container.

Also, are you doing any fiddling with the classpath in the startup of your container? Does it use builder slaves on the same machine? I suggest investigating these issues first.