简体   繁体   English

C#读取具有其他实例竞争的文本文件

[英]c# reading text file with contention from other instances

I have a app that reads data from a text file using: 我有一个应用程序,它使用以下命令从文本文件读取数据:

                CRD.reader = new StreamReader(fn,Encoding.UTF8,true,1024);
                CRD.readLine();

BUT I run 16 instances of this app in parallel on my 24 core machine. 但是,我在24核心计算机上并行运行了该应用程序的16个实例。 When I do this the total time taken is much greater than time it takes a single instance running on it's own (even though they are running in parallel). 当我这样做时,所花费的总时间要远远大于一个实例独自运行的时间(即使它们并行运行)。 I assume this is because of contention for the disk? 我认为这是因为争用磁盘?

I saw a suggestion for using a bufferedstream, but I don't understand how that differs from the code above. 我看到了使用缓冲流的建议,但是我不明白这与上面的代码有何不同。 Surely by specifying the buffer size as I have - I am using a "buffered" stream already? 当然可以通过指定缓冲区大小来确定-我已经在使用“缓冲”流了吗?

For my code, I have tried various different size for the buffer - but it does not appear to make much difference. 对于我的代码,我为缓冲区尝试了各种不同的大小-但这似乎没有太大的区别。

EDIT 1

If anyone could explain how a bufferedstream differs from what I am doing - that would be very helpful 如果有人可以解释缓冲流与我的操作有何不同,那将非常有帮助

EDIT 2

If I set a large buffer with 如果我用

CRD.reader = new StreamReader(fn,Encoding.UTF8,true,65536);
                CRD.readLine();

Can I force the whole buffer to be filled on first readLine? 我可以强制在第一个readLine上填充整个缓冲区吗? ie if my buffer > than filesize the whole file could/should be read into memory. 也就是说,如果我的buffer> than filesize可以/应该将整个文件读入内存。 It seems to me that the operating system works by allowing that much buffer, BUT not necessarily using it. 在我看来,操作系统可以通过允许这么多的缓冲区来工作,但不一定要使用它。

If file sizes, according to the comment, are about 2MB , the fast processing solution would be 根据评论,如果文件大小约为2MB ,则快速处理解决方案将是

  • first read completely into the memory, in one shot, using for example File.ReadAllText method 首先使用文件File.ReadAllText方法一次完整地读取到内存中

  • after process content, already present in memory , so it will be much faster, the reading line by line from the disk . 后处理的内容,已经存在于内存中 ,所以它会快很多 ,从线读线。

  1. Try to open file in read only mode 尝试以只读模式打开文件
  2. Try to use memory mapped file it could provide best performance for concurent file access 尝试使用内存映射文件,它可以为并发文件访问提供最佳性能

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM