[英]Efficient way to read first lines from several web files in C#
There is a web server that contains several long files (>500Kb) with different categories of news. 有一个Web服务器,其中包含带有不同新闻类别的多个长文件(> 500Kb)。 I only need to get the latest news in each category, first lines of the file (usually < 1Kb but can be more) and to make the download quick and because the connection is slow, my idea is to read line by line so downloading the minimum amount of data.
我只需要获取每个类别的最新新闻,即文件的第一行(通常<1Kb,但可以更多),并使下载速度更快,并且由于连接速度较慢,我的想法是逐行阅读,因此下载最小数据量。 At the moment I executing the below code but for the time it takes does not seem much improvement from downloading the full file.
目前,我执行以下代码,但是从下载完整文件来看,这似乎并没有太大的改善。
foreach(var newsType in newsTypes)
{
var request = WebRequest.Create("http://www.xxxx.com/" + newsType) as HttpWebRequest;
request.Timeout = 5000;
using (var response = request.GetResponse() as HttpWebResponse)
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
while (!reader.EndOfStream)
{
var fileRowCSV = reader.ReadLine();
...
if (old-news) break;
...
}
}
}
Is there and setting I need to modify so not all the file gets transferred? 是否存在需要修改的设置,因此并非所有文件都可以传输? Can I reuse one connection in some way that can read the other files?
我可以某种方式重用一个连接以读取其他文件吗? Any ideas on how to improve the process?
关于如何改进流程的任何想法?
Thanks 谢谢
I don't think there's a built-in way to do this in the .NET Framework. 我认为.NET Framework中没有内置的方法可以做到这一点。 However, if you're looking to simply speed up the entire process, you can (relatively easily, I think) parallelize the outer
foreach
loop. 但是,如果您只是想加快整个过程的速度,则可以(相对容易地,我认为)并行化外部
foreach
循环。
Alternatively, here is an MSDN example that displays the progress of downloading a file from the web. 另外, 这是一个MSDN示例,显示了从Web下载文件的进度。 You should be able to use the code in that example to accomplish what you want: after every X bytes, check if you've reached
old-news
and, if so, cancel the download. 您应该能够使用该示例中的代码来完成所需的工作:每X个字节之后,检查是否已收到
old-news
,如果是,则取消下载。 Fair warning, it's not just a few lines of code. 合理的警告,不仅是几行代码。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.