Efficient way to read first lines from several web files in C#

Question

There is a web server that contains several long files (>500Kb) with different categories of news. I only need to get the latest news in each category, first lines of the file (usually < 1Kb but can be more) and to make the download quick and because the connection is slow, my idea is to read line by line so downloading the minimum amount of data. At the moment I executing the below code but for the time it takes does not seem much improvement from downloading the full file.

foreach(var newsType in newsTypes)
{
    var request = WebRequest.Create("http://www.xxxx.com/" + newsType) as HttpWebRequest;
    request.Timeout = 5000;
    using (var response = request.GetResponse() as HttpWebResponse)
    {
        using (var reader = new StreamReader(response.GetResponseStream()))
        {
            while (!reader.EndOfStream)
            {
                var fileRowCSV = reader.ReadLine();
                ...
                if (old-news) break;
                ...
        }
    }
}

Is there and setting I need to modify so not all the file gets transferred? Can I reuse one connection in some way that can read the other files? Any ideas on how to improve the process?

Thanks

Answer 1

I don't think there's a built-in way to do this in the .NET Framework. However, if you're looking to simply speed up the entire process, you can (relatively easily, I think) parallelize the outer foreach loop.

Alternatively, here is an MSDN example that displays the progress of downloading a file from the web. You should be able to use the code in that example to accomplish what you want: after every X bytes, check if you've reached old-news and, if so, cancel the download. Fair warning, it's not just a few lines of code.