[英]Out of Memory Exception in Parallel.ForEach
I am using Parallel.ForEach to do my job but got "Out of Memory Exception". 我正在使用Parallel.ForEach来完成我的工作,但遇到了“内存不足异常”。
Parallel.ForEach(flist, (item) =>
{
string f1 = item.Split('|')[0];
string f2 = item.Split('|')[1];
a = File.ReadAllText(f1);
b = File.ReadAllText(f2);
Consume(a, b);
});
flist
's size is 351, a
and b
are string, each of them has 20kb size. flist
的大小为351, a
和b
为字符串,每个大小均为20kb。 At a certain time, the system memory blown up. 在特定时间,系统内存耗尽。
Consume return a string list, usually around 1000 strings in each iteration. 消费返回一个字符串列表,通常在每次迭代中大约有1000个字符串。
How to deal with it? 怎么处理呢?
Try replacing: 尝试更换:
Parallel.ForEach(flist, (item) =>
{
string f1 = item.Split('|')[0];
string f2 = item.Split('|')[1];
a = File.ReadAllText(f1);
b = File.ReadAllText(f2);
Consume(a, b);
});
With: 附:
Parallel.ForEach(flist,
new ParallelOptions { MaxDegreeOfParallelism = 4 },
(item) =>
{
string f1 = item.Split('|')[0];
string f2 = item.Split('|')[1];
a = File.ReadAllText(f1);
b = File.ReadAllText(f2);
Consume(a, b);
});
This will prevent too many threads being created. 这将防止创建太多线程。 Then you can always experiment with higher numbers and see if performance improves.
然后,您始终可以尝试使用更大的数字,看看性能是否有所提高。
You're reading an entire file into a single string twice per loop. 您正在将每个循环中的整个文件读入单个字符串两次。 That's very likely the source of your problem if the file is large.
如果文件很大,很可能是问题的根源。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.