Lucee sudden high Java CPU usage

Question

Running Lucee Lucee 5.2.7.62 on Ubuntu 16 / Java 8_181.

A couple of days ago, CPU usage started running at around 50%, which appears to be due to two Java threads belonging to Lucee spinning. Restarting Lucee and the O/S have no effect - the spin behaviour starts as soon as Lucee is running.

Drilling down with FusionReactor, the issue seems to relate to scheduled tasks. No changes to scheduled tasks seem to have taken place prior to this error. The server is in the UK time zone - I am wondering if the issue relates to switching from GMT to BST this weekend, though the problem started a couple of days before.

When I stack trace the spinning threads in Fusionreactor, I typically see something like this:

java.util.SimpleTimeZone.getOffsets(SimpleTimeZone.java:551)
- waiting on <0x5d7f0b89> (a java.util.SimpleTimeZone held by thread 84, Thread-50)
java.util.SimpleTimeZone.getOffset(SimpleTimeZone.java:540)
sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:293)
sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:236)
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2340)
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)
java.util.Calendar.setTimeInMillis(Calendar.java:1804)
java.util.GregorianCalendar.add(GregorianCalendar.java:1076)
lucee.runtime.schedule.ScheduledTaskThread.calculateNextExecution(ScheduledTaskThread.java:219)
lucee.runtime.schedule.ScheduledTaskThread._run(ScheduledTaskThread.java:121)
lucee.runtime.schedule.ScheduledTaskThread.run(ScheduledTaskThread.java:87)

Answer 1

Fusionreactor pointed to an issue with Scheduled Tasks with the Thread Visiualizer showing two spinning tasks, with stack traces similar to the below.

I killed these threads, and the spinning stopped. I could then see two of my scheduled tasks marked in pink as permanently stopped in the Lucee administrator. Re-enabling these processes and restarting Lucee brought the problem back, so I killed them again and again the tasks went pink in the Lucee Administrator. They did not run on their normal schedule either. Other scheduled tasks remained running OK, and after a few hours things were still normal.

I then removed and recreated the two scheduled tasks that seemed to be the issue, and restarted Lucee. The two tasks ran on schedule. I have therefore concluded that somehow the timing information for the two tasks had become corrupted, causing the spin when Lucee was trying to calculate the next run time. The Luce source code around the point it seemed to be spinning has a 'while(1)' loop which seemed to be incrementing a date variable - I suspect that's where things were stuck.

In summary, get shot of the scheduled tasks and recreate, and you may have a workaround.