Scheduling jobs with high CPU load in Quartz.NET

Question

I'm using Quartz.NET to schedule a job which loads a bunch of data from external sources and persists in a database. After downloading them some processing have to be on them too, which will create additional records in a different table in the database.

The downloader job is a stateful job and runs in every minute. The problem I've faced is that, after downloading the data the processing part of it could take much longer time than I expected.

How should I manage this? I've though about creating another job (which will run for one time only) when the data downloading finishes. In this case the downloader job can run in every minute (this was the original plan), because the download part takes 5-20 secs only, and the other job can process these records after the downloader one has finished. The processer job would fetch records marked as unprocessed from database and do the work on them.

Is this a correct approach to handle the processing? The other idea I came is to set up a WCF service which would process one downloaded element. This would be called on each downloaded element. However I don't think this would perform better than the other job approach.

Answer 1

Maybe a better approach should be using the job as an event, and have another service execute the download.

The job would just send a message defining the intent to another endpoint.

This is very common in an Event Driven Architecture with some sort of message broker or service bus, like NServiceBus or MassTransit.

That way your scheduler will remain lean and scalable.

Here is an example:

public class DownloadFileJob : IJob
{
    public IBus Bus { get; set; }
    public ILogger Logger{ get; set; }

    public void Execute(IJobExecutionContext context)
    {
        Bus.Send(new DownloadFileMessage());
        Logger.Info("Sending message requesting download of file.");
    }
}

Answer 2

You might look into setting up a JobListener for the download job. Simply create a class that implements the IJobListener interface, then put your processing code in the JobWasExecuted method:

public PostDownloadJobListener : IJobListener
{
    string Name { get { return "MyJobListener"; } }
    void JobToBeExecuted(JobExecutionContext context) { }
    void JobExecutionVetoed(JobExecutionContext context) { }
    void JobWasExecuted(JobExecutionContext context, JobExecutionException jobException)
    {
        // Perform processing here
    }
}

Register the listener using scheduler.AddJobListener(myJobListener); , and let the listener do the processing after the job successfully executes.

Answer 3

Your approach to having the download job schedule another job to do the processing is feasible and used commonly.

One thing to keep in mind though is that this works as long as the download job doesn't always schedule a processing job. If your download job always schedules a job (say every minute) and the scheduled job takes longer than that to run, you will eventually run out of processing threads and your download job will have to wait for a thread to be available.