Jenkins is running but service says it is failed

Question

Since I have installed jenkins 2.319.1 on CentOS7 via yum update jenkins I have the problem, that I can start jenkins as service via sudo systemctl start jenkins. Then jenkins will start an I can access it via the browser.. However if I try to get the status of the service via sudo systemctl status jenkins.

I get first:

PROD [root@abmrndplje001 grafra1969]$ systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
  Drop-In: /etc/systemd/system/jenkins.service.d
           └─00-proxy.conf
   Active: activating (start) since Wed 2021-12-15 19:42:42 CET; 1min 54s ago
     Docs: man:systemd-sysv-generator(8)
  Control: 969 (jenkins)
   CGroup: /system.slice/jenkins.service
           ├─ 969 /bin/sh /etc/rc.d/init.d/jenkins start
           ├─1004 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java ...
           ├─1035 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true...
           └─1041 /etc/alternatives/java -Djava.awt.headless=true -Xmx8g -XX:MaxPermSize=1024m -DJENKINS_H...

Dec 15 19:42:42 abmrndplje001.pass-consulting.com systemd[1]: Starting LSB: Jenkins Automation Server...
Dec 15 19:42:42 abmrndplje001.pass-consulting.com runuser[1004]: pam_unix(runuser:session): session open...0)
Dec 15 19:42:42 abmrndplje001.pass-consulting.com jenkins[969]: Starting Jenkins OpenJDK 64-Bit Server V....0
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: An illegal reflective access op...ed
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: Illegal reflective access by co...n)
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: Please consider reporting this ...$2
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: Use --illegal-access=warn to en...ns
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: All illegal access operations w...se
Hint: Some lines were ellipsized, use -l to show in full.
PROD [root@abmrndplje001 grafra1969]$

After about 5 min. The status changes from activating to failed and as reason is given a timeout. However the processes are still runing and jenkins is working and can be accessed via the browser.

PROD [root@abmrndplje001 grafra1969]$ systemctl status jenkins -l
● jenkins.service - LSB: Jenkins Automation Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
  Drop-In: /etc/systemd/system/jenkins.service.d
           └─00-proxy.conf
   Active: failed (Result: timeout) since Wed 2021-12-15 19:47:42 CET; 5min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 969 ExecStart=/etc/rc.d/init.d/jenkins start (code=killed, signal=TERM)
   CGroup: /system.slice/jenkins.service
           ├─1004 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Dj                                                                   ava.awt.headless=true -Xmx8g -XX:MaxPermSize=1024m -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenk                                                                   ins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --ht                                                                   tpListenAddress=localhost --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
           ├─1035 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -X                                                                   mx8g -XX:MaxPermSize=1024m -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/l                                                                   og/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --httpListenAddress=localhos                                                                   t --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
           └─1041 /etc/alternatives/java -Djava.awt.headless=true -Xmx8g -XX:MaxPermSize=1024m -DJENKINS_HOME                                                                   =/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cac                                                                   he/jenkins/war --daemon --httpPort=8080 --httpListenAddress=localhost --debug=5 --handlerCountMax=100 --handl                                                                   erCountMaxIdle=20

Dec 15 19:42:42 abmrndplje001.pass-consulting.com jenkins[969]: Starting Jenkins OpenJDK 64-Bit Server VM war                                                                   ning: Ignoring option MaxPermSize; support was removed in 8.0
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: An illegal reflective access operati                                                                   on has occurred
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: Illegal reflective access by com.goo                                                                   gle.inject.internal.cglib.core.$ReflectUtils$2 (file:/var/cache/jenkins/war/WEB-INF/lib/guice-4.0.jar) to met                                                                   hod java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: Please consider reporting this to th                                                                   e maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: Use --illegal-access=warn to enable                                                                    warnings of further illegal reflective access operations
Dec 15 19:42:55 abmrndplje001.pass-consulting.com jenkins[969]: WARNING: All illegal access operations will b                                                                   e denied in a future release
Dec 15 19:47:42 abmrndplje001.pass-consulting.com systemd[1]: jenkins.service start operation timed out. Term                                                                   inating.
Dec 15 19:47:42 abmrndplje001.pass-consulting.com systemd[1]: Failed to start LSB: Jenkins Automation Server.
Dec 15 19:47:42 abmrndplje001.pass-consulting.com systemd[1]: Unit jenkins.service entered failed state.
Dec 15 19:47:42 abmrndplje001.pass-consulting.com systemd[1]: jenkins.service failed.

How can one start the service correctly?

I just found out that the same jenkins version is running on different CentOS7 server and the service status is active.

The only difference I have seen so far, ist that the working version is not using runuser as the first command

[root@jenkins2-rnd grafra1969]# systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
   Active: active (running) since Thu 2021-12-16 15:57:56 CET; 18min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 4436 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)
  Process: 4448 ExecStart=/etc/rc.d/init.d/jenkins start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/jenkins.service
           ├─ 1883 ssh-agent
           ├─ 3329 ssh-agent
           ├─ 4452 /etc/alternatives/java -Djava.awt.headless=true -Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=f...
           ├─ 5150 ssh-agent
           ├─ 8365 ssh-agent
           ├─12083 sh -c ({ while [ -d '/var/lib/jenkins/workspace/Pam4Web_master@tmp/durable-8b6bf811' -a \! -f '/var/lib/jenkins/workspace/P...
           ├─12084 sh -c ({ while [ -d '/var/lib/jenkins/workspace/Pam4Web_master@tmp/durable-8b6bf811' -a \! -f '/var/lib/jenkins/workspace/P...
           ├─12085 sh -xe /var/lib/jenkins/workspace/Pam4Web_master@tmp/durable-8b6bf811/script.sh
           ├─12088 /usr/lib/jvm/java-11-openjdk/bin/java -classpath /var/lib/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven-3.6/boot...
           ├─12969 sleep 3
           ├─13279 ssh-agent
           ├─14595 ssh-agent
           ├─15882 ssh-agent
           └─17056 ssh-agent

Answer 1

I had exactly the same problem and fix it. The problem comes from your /etc/init.d/jenkins. You probably modify it and the update process (yum on rhel7) is not able to replace it. This 'old file' is not more compatible with the new version of jenkins. It's why systemctl commands are not working correctly.

Answer 2

The problem was that yum wanted to install a new version of /etc/init.d/jenkins. However since I hat edited it before. The new file has been installed in /etc/init.d/jenkins.rpmnew instead and systemd still used the old one. What I did was to take over my changes form the old /etc/init.d/jenkins to the new file. Then delete the old /etc/init.d/jenkins, rename /etc/init.d/jenkins.rpmnew to /etc/init.d/jenkins and restart the service.

Answer 3

We are experiencing the same problems... (on redhat 7)

The same issue exists on the jira from jenkins: https://issues.jenkins.io/browse/JENKINS-67334

It seems to be closed but I don't understand why...

Answer 4

i still did not understand what exactly need to be done:

"I had exactly the same problem and fix it. The problem comes from your /etc/init.d/jenkins. You probably modify it and the update process (yum on rhel7) is not able to replace it. This 'old file' is not more compatible with the new version of jenkins. It's why systemctl commands are not working correctly.'

can someone help?