Kotlin retry scheduled cron task if failed

Question

. 。 Answers to this question are eligible for a +100 reputation bounty. William Ross is looking for a : ：

I'm looking for the trade offs between using Spring Boot/Kotlin and Kube.netes CronJobs for scheduling jobs. I need maximum reliably and want to evaluate if I can do this with Spring Boot and Kotlin alone.

I have a Kotlin scheduling config file below. It has an important task scheduled to run at 11am each Monday.

What do I need to do for building resiliency or retry attempts in case the service is down at 11am?

Can these Spring Boot and Kotlin @Scheduled jobs be configured for enterprise level resiliency or do I need to look to use something like Kube.netes CronJobs to achieve this?

@Component
class CronConfig {

    private val logger = LoggerFactory.getLogger(CronConfig::class.java)

    // Run Monday morning @ 11am
    @Scheduled(cron = "0 0 11 * * MON")
    fun doSomething(){
        logger.info("Doing something")
    }
}

Answer 1

It's good that you're thinking about what might go wrong. (Too often we developers assume everything will go right, and don't consider and handle all the ways our code could fail!)

Unfortunately, I don't think there's a standard practice for this; the right approach probably depends on your exact situation.

Perhaps the simplest approach is just to ensure that your function cannot fail , by doing error-handling, and if needed waiting and retrying, within it. You could split the actual processing out to a separate method if that makes it more readable, eg:

@Scheduled(cron = "0 0 11 * * MON")
fun doSomethingDriver() {
    while (true) { // Keep trying until successful…
        try {
            doSomething()
            return // It worked!
        } catch (x: Exception) {
            logger.error("Can't doSomething: {}.  Will retry…", x.message)
            TimeUnit.SECONDS.sleep(10L)
        }
    }
}

fun doSomething() {
    logger.info("Doing something")
    // …
}

That's pretty straightforward. One disadvantage is that it keeps the thread waiting between retries; since Spring uses a single-threaded scheduler by default (see these questions ), that means it could delay any other scheduled jobs.

Alternatively, if your scheduled function doesn't keep retrying, then you'll need some other way to trigger a retry.

You could 'poll': store the time of the last successful run, change the scheduled function to run much more frequently, and have it check whether another run is needed (ie whether there's been no successful run since the last 11am Monday). This will be more complex — especially as it needs to maintain state and do date/time processing. (You shouldn't need to worry about concurrency, though, unless you've made the function @Async or set up your own scheduling config.) It's also a little less efficient, due to all the extra scheduled wake-ups.

Or you could trap errors (like the code above) but instead of waiting and retrying, manually schedule a retry for a future time, eg using your own TaskExecutor . This would also be more complex.