Inconsistent behavior of clustered Camel Quartz2 route

Question

We have one application developed using Spring(4.0.3.RELEASE) + Camel(2.13.4) + Quartz2 + Oracle11. We are deploying this application on Websphere 7 with JDK 1.6.

We have different environments with vertical and horizontal clustering. Where we are using two different machines, the clocks of the systems are in sync. The desired state is to have the job run only on one machine at a time. To achieve this, we have configured the environment as clustered using the following properties.

org.quartz.scheduler.instanceName = UAT_CLUSTER 
org.quartz.scheduler.instanceId = AUTO
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 1
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = 
org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = false
org.quartz.jobStore.dataSource = oracleDs                                                         
org.quartz.jobStore.isClustered = true
org.quartz.dataSource.oracleDs.jndiURL = jdbc/oracleDs

The behavior we are seeing is not consistent. In one environment, where we have two JVMs on the same machine , we are seeing that the QRTZ_TRIGGERS table has only one entry .

In another environments, where two machines with one JVM each , the behavior is sometimes similar and sometimes not. We see that the QRTZ_TRIGGERS table always has sometimes one and sometimes two entries .

In both the cases, the tables are updated as expected right at the time of trigger. However, the job doesn't run and the logs don't even have a trace.

The SCHED_NAME in the QRTZ_TRIGGERS table is appended with -camel-1-* . However, the behavior is not consistent. In one environment where we have 2 machines with one JVM each, we are seeing only one SCHED_NAME being added to the QRTZ_TRIGGERS table. In another environment with similar configuration, we are seeing that two SCHED_NAME are being added.

We would like to know the right and expected behavior of the tables and a way to achieve that. Any link to the documentation for the tables would be appreciated.

The route is simple and it is defined as

@Configuration
@ComponentScan(basePackages = { "com.company.app.package" })
public class BatchConfig extends SingleRouteCamelConfiguration {

@Value("#{ systemProperties['application.environment'] }")
private String appEnv;

@Override
public RouteBuilder route() {
    return new RouteBuilder() {
        public void configure() {
            from("quartzComponent://group/trigger?cron=0+0/5+*+?+*+*&recoverableJob=true").routeId("batchRoute")
                    .beanRef("firstProcessor").beanRef("secondProcessor").end();
        };
    };
}

@Override
protected void setupCamelContext(CamelContext camelContext) throws Exception {
    camelContext.addComponent("quartzComponent", quartzComponent());
}

@Bean
public QuartzComponent quartzComponent() {
    QuartzComponent quartz = new QuartzComponent();
    quartz.setPropertiesFile("env/" + appEnv + "/quartz.properties");
    return quartz;
}

Answer 1

I was able to fix this. The issue was with orphan threads. Even after shutdown of application, we were seeing that threads continue to run and QRTZ_SCHEDULER_STATE table would have all the previous instances since the restart of the server. If I delete those entries, the table would be updated with all the instances even after shutdown of application.

To overcome these, we made our root configuration file extend CamelConfiguration and used annotated Class that extends RouteBuilder and is scanned as Spring Component. Earlier we were using SingleRouteCamelConfiguration within our configuration.