Azure 上的 Lucee/Java 的 CPU 使用率为 100%

Question

On today's episode of wtf is going on.....

Issue we've been experiencing for the last few days. For some reason Java is getting pegged at 100% cpu usage. If we disable the lucee service, the cpu usage drops to a normal level. Once we enable it, the cpu usage immediately spikes up to 100%.

Full path is

/opt/lucee/jdk/jre/bin/java -Djava.util.logging.config.file=/opt/lucee/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

Our other servers running Lucee show this process to normally have around 1-3% cpu usage.

1. We've restarted Ubuntu multiple times to no avail.
2. We've disabled all ports in case there was some weird traffic causing issues, again to no avail.
3. We've verified there are no lucee scripts tasks running.
4. We do have a separate SQL Service instance running with Microsoft which does not show any weird DB usage. We also have around 5 other servers which access that same DB and which are not experiencing this issue.
5. I've downgraded Lucee from 5.3.4.80 to 5.2.9.3, again no luck

Other info for the server

OS      Linux (4.4.0-174-generic) 64bit
Servlet Container   Apache Tomcat/8.5.6
Java    1.8.0_112 (Oracle Corporation) 64bit

This originally happened to us about two weeks ago. We came in on Monday 2/24, and by about 9am CT, one of our servers started experiencing this issue. We setup a separate Azure VM and copied all our files over and got everything up and running just fine. Now, two weeks later two other servers are starting to have this same issue.

Appreciate any help you guys can provide.

Answer 1

I had the same issue on an AWS server running Ubuntu and Java 8_181. It started suddenly in the middle of the night. Top was showing 2 CPU's fully loaded, just like yours. Restarting Lucee/Tomcat and rebooting had no effect.

Fusionreactor pointed to an issue with Scheduled Tasks with the Thread Visiualizer showing two spinning tasks, with stack traces similar to the below.

I killed these threads, and the spinning stopped. I could then see two of my scheduled tasks marked in pink as permanently stopped in the Lucee administrator. Re-enabling these processes and restarting Lucee brought the problem back, so I killed them again and again the tasks went pink in the Lucee Administrator. They did not run on their normal schedule either. Other scheduled tasks remained running OK, and after a few hours things were still normal.

I then removed and recreated the two scheduled tasks that seemed to be the issue, and restarted Lucee. The two tasks ran on schedule. I have therefore concluded that somehow the timing information for the two tasks had become corrupted, vausing the spin when Lucee was trying to calculate the next run time. The Luce source code around the point it seemed to be spinning has a 'while(1)' loop which seemed to be incrementing a date variable - I suspect that's where things were stuck.

In summary, get shot of the scheduled tasks and recreate, and you may have a workaround.

java.util.SimpleTimeZone.getOffsets(SimpleTimeZone.java:551) - locked <0x5d7f0b89> (a java.util.SimpleTimeZone) java.util.SimpleTimeZone.getOffset(SimpleTimeZone.java:540) sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:293) sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:236) java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2340) java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312) java.util.Calendar.setTimeInMillis(Calendar.java:1804) java.util.GregorianCalendar.add(GregorianCalendar.java:1076) lucee.runtime.schedule.ScheduledTaskThread.calculateNextExecution(ScheduledTaskThread.java:219) lucee.runtime.schedule.ScheduledTaskThread._run(ScheduledTaskThread.java:121) lucee.runtime.schedule.ScheduledTaskThread.run(ScheduledTaskThread.java:87)