Jenkins：队列中的作业被卡住，没有被触发重新启动

Question

For a while, our Jenkins experiences critical problems. We have jobs hung, our job scheduler does not trigger the builds. After the Jenkins service restart, everything is back to normal, but after some period of time all problem are return. (this period can be week or day or ever less). Any idea where we can start looking? I'll appreciate any help on this issue

Answer 1

Muatik has made a good point in his comment, the recommended approach is to run jobs on agents (slave) nodes. If you already do it, you can look at:

1. Jenkins master machine CPU, RAM and hard disk usage. Access the machine and/or use plugin like Java Melody . I have seen missing graphics in the builds test results and stuck builds due to no hard disk space. You could also have hit the limit of RAM or CPU for the slaves/jobs you are executing. You may need more heap space .
2. Look at Jenkins log files, start with severe exceptions. If the files are too big or you see logrotate exceptions, you can change the logging levels, so that fewer exceptions are logged. For more details see my article on this topic . Try to fix exceptions that you see logged.

3. Go through recently made changes that can be the cause of such behavior, for example, new plugins, changes to config files (jenkins.xml)?

4. Look at TCP connections. Run netstat -a Are there suspicious connections (CLOSED_WAIT status)?

5. Delete old builds that you do not need.

Answer 2

We have been facing this issue from last 4 months, and tried everything, changing resources CPU & memory, increasing desired nodes in ASG. But nothing seems worked.

Solution: 1. Go to Manage Jnekins-> System Configurationd-> Maven project configurations 2. In "usage" field, select "Only buid Jobs with label expressions matching this nodes"

Doing this resolved it and jenkins is working as a Rocket now:)