为什么我的Spring @Scheduled任务停止？

Question

I'm using Spring's @Scheduled feature to run a task every 8 hours every day in a WAS 8.5 environment using Spring 3.1.1. It starts and runs great for awhile, then stops for no apparent reason. There's nothing in my logs to indicate a failure. Any idea what might make this occur? It's happened a couple of times now. It's not due to server restarts, etc.

package com.my.project.scheduler;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.List;
import java.util.concurrent.TimeUnit;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.sql.DataSource;

import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.ApplicationContext;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.web.context.ContextLoaderListener;

import com.my.project.scheduler.model.RetransmitEvent;

public class RetransmitTimer {    

    private static final Logger LOG = Logger.getLogger( RetransmitTimer.class );

    @Scheduled( cron = "${retransmit.cron.interval.1}" )
    public void retransmitSchedule1() {
        retransmit();
    }
    @Scheduled( cron = "${retransmit.cron.interval.2}" )
    public void retransmitSchedule2() {
        retransmit();
    }
    @Scheduled( cron = "${retransmit.cron.interval.3}" )
    public void retransmitSchedule3() {
        retransmit();
    }
    private void retransmit() {
        LOG.info( "############# AUTOMATIC RETRANSMIT EXECUTING ##############" );        
        try
        {
            [code to gather retransmitEvents omitted]
            if ( retransmitEvents.size() > 0 ) {
                LOG.info( "Total retransmits found=" + retransmitEvents.size() );
                submitRetransmits( retransmitEvents );
            }
        }    
        finally
        {    [method cleanup, etc]
            LOG.info( "############# AUTOMATIC RETRANSMIT COMPLETED ##############" );
        }
    }

    private void submitRetransmits( List<RetransmitEvent> retransmitEvents ) {
        ApplicationContext context = ContextLoaderListener.getCurrentWebApplicationContext();
        DataSource ds1 = (DataSource) context.getBean("jdbc/DS1");
        DataSource ds2 = (DataSource) context.getBean("jdbc/DS2");
        [rest of the code to process database updates omitted]
    }
}

My applicationContext.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:task="http://www.springframework.org/schema/task"
    xmlns:p="http://www.springframework.org/schema/p"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
                        http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
                        http://www.springframework.org/schema/context
                        http://www.springframework.org/schema/context/spring-context.xsd
                        http://www.springframework.org/schema/task
                        http://www.springframework.org/schema/task/spring-task-3.0.xsd">

    <context:property-placeholder location="classpath:my-scheduler.properties" />
    <context:annotation-config/>
    <context:component-scan base-package="com.my.project" />

    <task:annotation-driven/> <!-- for the Scheduled annotations in the timer code -->

    <bean id="retransmitTimer" class="com.my.project.scheduler.RetransmitTimer" scope="singleton"/>

    <!-- The following Spring beans replace what would normally go into the web.xml file -->
    <!-- They're for the attachmentViewer.jar attachment retry process -->
    <bean id="jdbc/DS1" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiName" value="jdbc/DS1" />
        <property name="lookupOnStartup" value="false" />
        <property name="cache" value="true" />
        <property name="proxyInterface" value="javax.sql.DataSource" />
    </bean>

    <bean id="jdbc/DS2" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiName" value="jdbc/DS2" />
        <property name="lookupOnStartup" value="false" />
        <property name="cache" value="true" />
        <property name="proxyInterface" value="javax.sql.DataSource" />
    </bean>
</beans>

my-scheduler.properties

# Retransmit preferences
#   CRON parms: Second Minute Hour Day-of-Month Month Day-of-the-Week
#   So, we're set to run everyday at the top of the hour at 4am, 12pm and 8pm
retransmit.cron.interval.1=0 0 4 * * ?
retransmit.cron.interval.2=0 0 12 * * ?
retransmit.cron.interval.3=0 0 20 * * ?

I know I could have setup the CRON parm for every 8 hours on one line, but users can get very creative in what they want, so I left the 3 entries in.

Here's the log snippets to show that it was working for several days. The server was restarted on 3/4 in the afternoon and hasn't been restarted since.

[2016-03-04 20:00:00,047] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-04 20:00:01,853] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-05 04:00:00,111] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-05 04:00:00,531] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-05 12:00:00,167] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-05 12:00:00,172] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-05 20:00:00,197] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-05 20:00:00,312] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-06 04:00:00,200] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-06 04:00:00,438] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-06 12:00:00,193] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-06 12:00:00,198] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-06 20:00:00,193] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-06 20:00:00,565] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-07 04:00:00,201] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-07 04:00:00,207] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-07 12:00:00,204] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-07 12:00:00,260] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-07 20:00:00,197] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-07 20:00:01,405] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############
[2016-03-08 04:00:00,198] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT EXECUTING ##############
[2016-03-08 04:00:00,568] [pool-7-thread-1] INFO  com.excellus.blue2.scheduler.RetransmitTimer - ############# AUTOMATIC RETRANSMIT COMPLETED ##############

No more entries after 3/8 4am. I should also note that there were no retransmit entries found during this period that would call sub method submitRetransmits().

Any idea why these scheduled tasks would stop? Thanks in advance.

Answer 1

Try to send kill -3 to see the stack trace. Probably some process is stack and cannot continue. You can also check if there are some timeouts on your transmission, if not it's better to set it.

Answer 2

Turned out my scheduler application was running all the while. It was a logging issue. I separated the logging from other projects and it revealed it was still running OK. Thanks for the all suggestions.