Strategy for managing long running processes in a web app?

Question

I am developing a web asp.net application. A feature of the application is that the user is able to queue up a number of long running operations, then hit run the queue will be processed.

I don't want the user to have to sit and wait for all the operations to complete or indeed keep the browser / application open. However should they remain or return to the page I would like them to be able to see the status of each job ie Waiting, in progress, completed or failed.

I am naturally looking for the most robust, reliable and scalable solution as potentially there maybe a great number of jobs in the queue at any time.

From my research it has been suggested that the asp.net could call into windows service? That perhaps is hosting a WCF web service.

Looking for any advice from anyone who might have had a similar requirement in the past.

Answer 1

Also check out MSMQ,

http://msdn.microsoft.com/en-us/library/ms711472(v=vs.85).aspx

Basically the web application places the "job" or "group of jobs" into a queue.

The service runs constantly, dequeueing the next "job" or "group of jobs" and getting down to business with them.

When completed, the service puts the results into another queue or in a DB (or some other shared resource) that the web application accesses and displays the status of, or the results of.

Enjoy.

Answer 2

A usually good approach is to have a Windows Service that performs scheduled jobs. You can use a library like Quartz.NET which has all kind of scheduling features. If you don't need specific scheduling, you can just have the Windows Service act as the worker process and simply consume the jobs. In addition to that you need to facilitate communication between your web application and the service that is performing the queued up tasks. You can use a number of techniques for this, such as using a table in the database or just using some other kind of inter process communication technique (remoting, named pipes, sockets, message queues etc).

I actually implemented something like this, where there is a Windows Service performing scheduled jobs and there is an ASP.NET web application that is putting up all the requests for the jobs. The ASP.NET application posts some jobs into the database and these are consumed later on by the service process.

Answer 3

Using windows service is a good idea. Though, I don't have any experience using it and don't know about unexpected pitfalls.
Also, you might want to check out the Fire And Forget pattern. (More info and samples can be found here ) Using multi-threading could be another option. For more info, check out:
http://msdn.microsoft.com/en-us/magazine/cc163587.aspx
http://msdn.microsoft.com/en-us/magazine/cc164128.aspx#S6
http://msdn.microsoft.com/en-us/library/ff647332.aspx

Answer 4

I have a similar problem. I have a web app that kicks off one long running process. I'd like to update the UI so the user see's what is going on. I have a class that gets some properties set and then starts the process. Here is my approach.

1) add a property to my class, job id.
2) when the web user starts the job, generate a new job id and assign it to the property above.
3) as my class is running its course, update a log table in a db. update it based on job id.
4) on the front end, have some simple ajax process check the status of the job id ever x seconds.

That's it.