Jenkins headless linux slave connection error

Question

I get java.net.SocketException: Connection reset when trying to connect a headless debian slave to my main Jenkins master. It's been running good for a couple of weeks without error and suddenly I can't get it to connect.

I can telnet into the master on port 6256.

Slave output:

#> java -jar ~/slave.jar -jnlpUrl https://test.tv/jenkins/computer/debian_slave_node_1/slave-agent.jnlp
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Hudson agent is running in headless mode.
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://test.tv/jenkins/, http://test.tv/jenkins/]
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to test.tv:6256
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 12, 2013 10:10:46 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: Connection reset
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:189)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    at hudson.remoting.ClassicCommandTransport.create(ClassicCommandTransport.java:98)
    at hudson.remoting.Channel.<init>(Channel.java:391)
    at hudson.remoting.Channel.<init>(Channel.java:387)
    at hudson.remoting.Channel.<init>(Channel.java:348)
    at hudson.remoting.Channel.<init>(Channel.java:344)
    at hudson.remoting.Channel.<init>(Channel.java:332)
    at hudson.remoting.Engine.run(Engine.java:238)

Jenkins Master output:

java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Answer 1

Hard to say what happens before knowing more. In particular:

• does it fail at every connection request ? If so please check your auth.log on your server as it seems the failure happens after "status INFO: Handshaking" and before "Connected"
• are your clock properly synchronized ?
• what OSes are you using ? Which java version on your slaves ? check your PATH and JAVA_HOME
• have you updated jenkins lately ? If so, have you updated the slave(s) ? Also have you tried reverting the update ? Which version are you using ? 1.520 took out support for JDK 1.5.
• how do you run your java command ? From a terminal in an SSH session ? Is this scripted from the master ?

See also Jenkins slave jobs failing on "Unexpected termination of channel"

Answer 2

I've run into the same issue, which seemed to be worked-around by running jenkins-slave as root. Now, the thing periodically disconnects with that very same error, which I'm wondering is due to a LB keepalive. I'll update this answer as soon as I get more info.

Answer 3

In my case everything was working under root account. But under user account, node client failed to connect to server with message "failed Unexpected termination of the channel". The problem was with access-or-write to working folder. As soon as I've reassigned owner throug "chown -R", node starts successfully.