设置ThreadPool.MaxThreads后从多个线程读取文件时出现WebException（超时）

Question

So, I tracked down the issue, but I don't understand the root cause and I'm curious.

I have multiple threads reading files (sometimes the same file, but usually different files. This doesn't seem to matter) from a local drive. This is the test setup, but in production these files are retrieved from a web server.

Anyway, I noticed that, after calling ThreadPool.SetMaxThreads() , I was receiving timeouts reading these files. Removing that line makes the problem go away. My hunch is that it has to do with setting the number of asynchronous IO threads ( completionPortThreads , the second argument), but even when I set that value to a large number (50, 100, ...), the issue remains.

Removing the call to SetMaxThreads "fixes" the issue, though it means I can't increase or decrease the number of threads for testing purposes.

Here is a block of code which reproduces the issue. The file size doesn't matter as my test files range anywhere from 2KB to 3MB.

class Program
{
    static void Main(string[] args)
    {           
        _count = 15;
        // Comment this line out and everything works
        ThreadPool.SetMaxThreads(13, 50);
        using (var mre = new ManualResetEvent(false))
        {
            for (int i = 0; i < _count; ++i)
            {
                ThreadPool.QueueUserWorkItem(ThreadFunc, mre);
            }

            mre.WaitOne();
        }
    }

    private static readonly ConcurrentStack<byte[]> _files = new ConcurrentStack<byte[]>();
    private static int _count;

    private static void ThreadFunc(object o)
    {       
        const string path = @"SomeLocalFile";
        var file = ReadFile(path);
        _files.Push(file);
        if (Interlocked.Decrement(ref _count) == 0)
        {
            ((ManualResetEvent)o).Set();
        }
    }

    private static byte[] ReadFile(string uri)
    {
        var request = WebRequest.Create(uri);

        using (var response = request.GetResponse())
        using (var stream = response.GetResponseStream())
        {
            var ret = new byte[stream.Length];
            stream.Read(ret, 0, ret.Length);
            return ret;
        }
    }       
}

So, yeah, not sure what's going on here. Even with a large value for IO threads I timeout on each test. I'm certainly missing something.

Answer 1

FileWebRequest which is the type returned by WebRequest.Create() also uses ThreadPool.QueueUserWorkItem. Since you limit the worker threads, the queued work of FileWebRequest never gets executed. You need to set max worker threads to at least _count + 1 (plus 1 so that there is at least one thread the can process the queued work by FileWebRequest).

FileWebRequest.GetRequestStream does the following:

1. ThreadPool.QueueUserWorkItem(read file)
2. Wait until the file is read or timeout is reached

Better Solution : Do not enqueue items to the ThreadPool. Use WebRequest.GetResponseAsync instead.