Why does Quartz .NET 3.0.7 scheduler hangs on DB unavailable error

Question

I'm having an issue with a very simple non-clustered quartz server within a C# console application (run as windows service). It includes a single scheduler, that reads from and writes to a single database, meaning the same group of "qrtz_*" tables. Occasionally, mostly at the weekend at night, when the company I work at performs DB backups and general maintenance, the scheduler listener I set up throws

    QuartzSchedulerListener

Scheduler Error -> MESSAGE: An error occurred while scanning for the next trigger to fire.

Couldn't acquire next trigger: Fatal error encountered during command execution.

at Quartz.Impl.AdoJobStore.JobStoreSupport.d__229.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 2688 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Quartz.Impl.AdoJobStore.JobStoreSupport.d__262`1.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 3816 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Quartz.Impl.AdoJobStore.JobStoreSupport.d__262`1.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 3831 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Quartz.Impl.AdoJobStore.JobStoreSupport.d__228.MoveNext() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 2542 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Quartz.Core.QuartzSchedulerThread.d__28.MoveNext() in C:\projects\quartznet\src\Quartz\Core\QuartzSchedulerThread.cs:line 262

I'm assuming due to DB temporarily unavailable.

Then, sometimes, it just hangs and stops executing any trigger - it doesn't update triggers' NEXT_FIRE_TIME and there's nothing I can do but restarting the console app (reconfiguring the scheduler and restarting it).

I've tried increasing dbCOmmandTimeout in the scheduler config connection string but it just looks like it gets stuck on some kind of deadlock.

PS: I have a similar setup, a group of quartz 2.6.1 schedulers reading/writing from several DBs. They get the same errors on DB maintenance but they have never stopped in more then a year of activity.

I've run out of ideas, please let me know whether this occurred to anyone.

Thank you

Answer 1

We had the same problem with Quartz version < 3.1.. There's a changelog: https://github.com/quartznet/quartznet/blob/main/changelog.md

I think you have to update the Quartz to at least 3.1 where some deadlock problem on DB unavailability has been fixed:

There is also a very important bug fix present for lock handling on retries. There was a possibility for a deadlock in database lock handling in some situations.

Hope it will help as it helped me.