使用 spring 和长时间运行的线程正常关闭

Question

We have the problem that there are multiple async threads that process a bunch of data and need minutes or some of them even hours to end. The pod also processes some other short running request.

It might occur that the pod has to be moved, and a shutdown will happen during this processing. In this case, we want the threads to finish their work in short term and leave a status, which we can use to process the remaining data.

We already use graceful.shutdown=enable .

The idea now is that we introduce a method with @PreDestroy in the bean that creates the async threads. When this method is called, it will set a "shutdown" flag in a different bean. All the long-running threads check this flag during processing and will stop processing when it's true and write a clean state of their processing to the database.

This is more or less working... but not all the time.

As I understood after getting the shutdown trigger at first, there is a configurable time spring.lifecycle.timeout-per-shutdown-phase where threads can continue processing their work without reduction in any of the needed resources. After the time is over, all the shutdownhooks are processed in an unknown order. This brings me to the thought that I might not have all the necessary resources for leaving a clean state, when I use the approach with the @preDestroy and the flag.

Is there a better solution to this?

Is there a need to add some more configuration to the threads like setAwaitTerminationSeconds(60); or setWaitForTasksToCompleteOnShutdown(true); ?

Answer 1

What I found is, when using graceful.shutdown=enable and @PreDestroy in spring, that you need to synchronize the run() method and the @PreDestroy method (provided that both are in the same class):

@Service
public class MyClass implements Runnable {
  private volatile boolean shutDown = false;
  public MyClass( TaskScheduler taskScheduler) {
        taskScheduler.scheduleWithFixedDelay(this, 100);
  }
  @PreDestroy
    public synchronized void preDestroy() {
        this.shutDown = true;
        log.info("@PreDestroy Callback triggered for shutting down");
    }
 @Override
    public synchronized void run() {
        if (this.shutDown) {
            this.shutdownCounter++;
            return;
        }
        // put in here, whatever you long running task needs to do
    }

And you need to make sure, that your run() method is not doing anything, when preDestroy() was already called, hence the shutdown - flag.

What the synchronize does is, that preDestroy() automatically waits until run() is finished regularly. Only then preDestroy() is called, and by setting the shutDown flag, it prevents that it starts another long running thread, even in case run() is called again. Instead run() returns immediately.

Because otherwise, when you don't synchronized, then the preDestroy() is called immediately after the spring - server get a shutdown signal but spring thinks if preDestroy() has run and finished successful, that it can shut down itself immediately. Which is not what you want, because then all run() threads gets interrupted.

I also found, that this example above does not work when using the @Scheduled - annotation (instead of calling taskScheduler.scheduleWithFixedDelay(this, 100) in the constructor)