在IIS 7中使用TPL-处置任务/线程/请求

Question

Forgive the title, I am not quite sure how to word what I am looking for, or if I even know what I am looking for.

I have a service that is running that takes a potentially large amount of requests concurrently. Additionally, within the service, I must make a certain amount (usually 10-15) requests to external services, which I am running in parallel.This code looks like the following

        Task<IResponse>[] tasks = new Task<IResponse>[Adapters.Count];
        for(int i = 0; i < Adapters.Count;i++)
        {
            IAdapter adapter = Adapters[i];

            Func<IResponse> makeExternalHttpRequest= () => adapter.MakeExternalHttpRequest(element, mappings);
            tasks[i] = Task.Factory.StartNew<IResponse>(() =>
                {
                    try
                    {
                        var result = makeExternalHttpRequest();
                        if (!token.IsCancellationRequested)
                        {
                            return result;
                        }
                    }catch(Exception exception)
                    {

                    }

                    return null;

                }, token);
        };
        var timeout = ...some timeout value
        Task.WaitAll(tasks,timeout, token);
        tokenSource.Cancel();

        for (int i = 0; i < tasks.Length; i++)
        {
            if (tasks[i].Result != null)
            {
                if (tasks[i].IsCompleted)
                {
                    results.Add(tasks[i].Result);
                }
            }
        }

Everything seems to be working as I expected for the past 6-7 months, and then yesterday, the server went down and I was sent the following from our system admin.

If you notice the highlighted area, it is quite large for the time elapsed.

I am guessing this contributed to why the server went down, but we are still looking into what happened.

Any idea on what is going on, and what my next steps should be?

Answer 1

I really think your next steps should be to gather more information. The way the code is written, your request is going to return right away after the timeout expires (or when all tasks have completed, whichever comes first). The other tasks are going to run to completion on their own, in the background, and assuming that the makeExternalHttpRequest() method times out eventually, they will all spin down in their own time.

Your cancellation token is not doing anything useful, as the code is going to block on the line above it.

What I would do is figure out which of the external requests is taking the longest, and maybe log a warning when that happens. Then at least you will have better information as to what is happening. See this pesudo-code:

(somewhere up above, create a logger and a System.Diagnostics.Stopwatch

              try
                {
                    var result = makeExternalHttpRequest();
                    if(stopwatch.Elapsed > SomeUnreasonableTimespan) 
                          logger.Warn("Task exceeded reasonable execution period." + TaskData);

                    if (!token.IsCancellationRequested)
                    {
                        return result;
                    }
                }catch(Exception exception)
                {

                }

                return null;

As a general comment, I find this piece of information most useful:

Everything seems to be working as I expected for the past 6-7 months, and then yesterday, the server went down and I was sent the following from our system admin.

This means that the problem is probably not related to your code, but related to something else. When I troubleshoot, I tend to ignore things that haven't changed and focus on things that have. Servers are going to crash from time to time. The thing you need to worry about is are they meeting your requirements most of the time? If so, then I wouldn't worry about this code too much.