When using Oracle JDK 1.8.0_60 on two centos 7.1 systems one Jenkins slave fails to collect artifacts with NoClassDefFoundError

Question

Riddle me this:

How can two nearly identical jenkins slaves running with Oracle JDK 1.8 u60 Java have one fail to collect artifact with a classdef error while the other is fine? In both case the following are identical

• system information shown from jenkins
• boot classpath
• jdk
• jdk files (these are deployed using SVN so we know they are identical)
• yum installed (mostly see below)

Note I can replicate this by going to script console and running the following

import org.apache.tools.ant.Location
Location l = new Location()

works on the good and fails in same way as a real build on the bad.

Error

java.io.IOException: remote file operation failed:... at 
hudson.remoting.Channel@35f2fb2:linengbld50: java.io.IOException: 
Remote call on linengbld50 failed
...
Caused by: java.io.IOException: Remote call on linengbld50 failed
...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
 org.apache.tools.ant.Location
...

Yum difference

Good v Bad

• hiera.noarch 1:1.3.4-5 vs 1.3.4-1
• net-snmp-* 1:5.7.2-24.el7_2.1 vs 1:5.7.2-24.el7
• ruby-augeas 0.5.0-1 vs 0.4.1-3
• zabbix 2.4.8 vs 2.4.7

Only ON Bad

• python-chardet.noarch 2.2.1-1.el7_1
• python-kitchen.noarch 1.1.1-5.el7
• lsof
• yum-utils.noarch 1.1.31-34.el

Answer 1

Using Script console was a great debugging tools as it executes in the right environment. getting output of which java on each node showed a difference. @mmasi found that bad showed a path whereas good returned null.

Even though we run slave using specific JDK java it used first java found on path for its activities (archiving).

• modify alternatives to set java to 1.8 (centos)

  sudo alternatives --install /usr/bin/java java \\ /opt/tools/Java/jdk1.8.0_60/bin/java 1;java -version

• Disconnect and reconnect node (restart the slave)

• Repeat the tiny script console check OK

• Repeat Build test (tiny build that archives a file) OK