具有实时接口进度状态的Ajax服务器进程？

Question

I have a process that retrieves html from a remote site and parses it. I pass several URL's into the method, so I would like to ajaxify the process and give a screen notification each time a URL completes parsing. For example, this is what I am trying to do:

List<string> urls = ...//load up with arbitary # of urls

foreach (var url in urls)
{
    string html = GetContent(url);
    //DO SOMETHING
    //COMPLETED.. SEND NOTIFICATION TO SCREEN (HOW DO I DO THIS)
}

public static string GetContent(string url)
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "GET";

    using (var stream = request.GetResponse().GetResponseStream())
    {
        using (var reader = new StreamReader(stream, Encoding.UTF8))
        {
            return reader.ReadToEnd();
        }
    }
}

In each iteration in the loop, I want to show the URL was completed and moving on to the next one. How can I accomplish this?

Answer 1

The first thing you need to worry about is the fact (I'm assuming) that you're running a potentially long-running operation in ASP.NET code. This will become a problem when you run in to IIS timeouts. (By default, 90 seconds.) Assume you're processing ten URLs, each of which takes 15 seconds to complete reader.ReadToEnd() – your code will time out and get killed after the sixth URL.

You might be thinking "I can just crank up the timeout," but that's not really a good answer; you're still under time pressure.

The way I solve problems like this is to move long-running operations into a stand-alone Windows Service, then use WCF to communicate between ASP.NET code and the Service. The Service can run a thread pool that executes requests to process a group of URLs. ( Here's an implementation that allows you to queue work items.)

Now, from your web page, you can poll for status updates via AJAX requests. The handler in your ASP.NET code can use WCF to pull the status information from the Service process.

A way to do this might be to assign each submitted work unit a unique ID and return that ID to the client. The client can then poll for status by sending an AJAX request for the status of work unit n . In the Service, keep a List of work units with their statuses ( lock ing it as appropriate to avoid concurrency problems).

public class WorkUnit {
   public int ID { get; set; }
   public List<string> URLs { get; set; }
   public int Processed { get; set; }
}

private var workUnits = new List<WorkUnit>();

private void ExecuteWorkUnit(int id) {
    var unit = GetWorkUnit(id);

    foreach (var url in unit.URLs) {
         string html = GetContent(url);
         // do whatever else...
         lock (workUnits) unit.Processed++;
    }
}

public WorkUnit GetWorkUnit(int id) {
    lock (workUnits) {
         // Left as an exercise for the reader
    }
}

You'll need to fill in methods to add a work unit, return the status of a given work unit, and deal with the thread pool.

I've used a similar architecture with great success.