[英]Is there a limiltation of simultaneous filestreams?
I have a strategic question to the use of simultaneously opened fstream
s. 对于同时打开的fstream
的使用,我有一个战略性的问题。 I have to write a program which has too read a large amount of files. 我必须编写一个程序,该程序也读取了大量文件。 In each file there are information to a bunch of identifiers, but one time. 每个文件中都有一堆标识符的信息,但是只有一次。 I have to compute this information and than save it for each identifier in a separate file. 我必须计算此信息,然后将每个标识符的信息保存在单独的文件中。 Every identifier appears in several files and should be saved every time in the same file (One identifier with many times). 每个标识符都出现在多个文件中,并且每次应保存在同一文件中(一个标识符多次)。 I expect some hundred identifiers so I doubt I should have several hundred filesteams open simultaneously. 我希望有数百个标识符,所以我怀疑我是否应该同时打开数百个文件组。
So is there a limitation of simultaneous filestreams? 那么同步文件流是否有限制? Or do you propose another way of doing this? 还是您提出另一种方法?
The program will compute a massive amount of data (about 10GB or larger) and perhaps computes several hours. 该程序将计算大量数据(大约10GB或更大),并可能计算几个小时。
Thanks 谢谢
There's ultimately a limit to anything. 最终任何事物都有极限。 Files are a perfect example of something managed by the operating system, and you will have to consult your OS documentation for the specific limit. 文件是由操作系统管理的文件的完美示例,并且您必须查阅OS文档以获取特定限制。 In Linux, I believe it is configurable in the kernel. 我相信在Linux中,它是可在内核中配置的。 There may additionally be user and process quotas. 可能还会有用户和进程配额。
I don't think 200 is too many to ask. 我认为200个要求不高。
It's quite simple to try and see. 尝试看很简单。 Just write a program that keeps opening more files until you get an error. 只需编写一个程序,该程序将不断打开更多文件,直到出现错误。
On Mac OS X 10.8, this program 在Mac OS X 10.8上,此程序
#include <iostream>
#include <fstream>
#include <iomanip>
#include <string>
int main() {
int i = 0;
std::ofstream *f;
do {
f = new std::ofstream( std::to_string( i ++ ) );
} while ( * f << "hello" << std::flush );
-- i; // Don't count last iteration, which failed to open anything.
std::cout << i << '\n';
}
Produces the output 253
. 产生输出253
。 So if you're on a Mac, you're golden :) . 因此,如果您使用的是Mac,那么您会很高兴:)。
The C++ standard does not define a limit for how many (or how few, I believe, but I haven't looked) files you can have open at the same time. C ++标准没有定义可以同时打开的文件数量限制(我相信,但我没有看过)。
A particular implementaton of a C++ library may have a limit (which may or may not be documented). C ++库的特定实现可能有一个限制(可能会或可能不会有记录)。 The operating system will most likely have some limit for the whole system, and another limit per process. 操作系统很可能对整个系统有一些限制,而每个进程都有另一个限制。 What those limits are will vary, so there's no easy way to tell. 这些限制是什么会有所变化,因此没有简单的方法可以说出来。 And they may also be artificially lowered by various settings that the system owner configures. 而且,系统所有者还可以通过各种设置来人为地降低它们。
And even if you know what all those limits are, there could be dynamic limits that vary depending on the circumstances - for example, if the whole system allows 16384 files open, the per process limit is 1000, and the C++ library allows 1024, you may not be able to open a single file, because there is no memory available for the OS to allocate some critical block of data. 而且即使您知道所有这些限制是什么,也可能会根据情况而定有动态限制-例如,如果整个系统允许打开16384个文件,每个进程的限制为1000,而C ++库允许1024,则可能无法打开单个文件,因为操作系统没有可用的内存来分配一些关键数据块。
There is no limit on the fstreams you can open simultaneously, however, your os limits the number of files that can be opened at the same time. 您可以同时打开的fstream没有限制,但是,您的操作系统会限制可以同时打开的文件数。 Althought some hundreds files doesn't seem to be too much for a general os, I would suggest you to read all the information beforehand (possibly opening several files at a time, but considering the possibility of a call to "open" to fail, in which case you should try again after closing some of the previously opened files) then do the processing and store the results on some internal data structure. 虽然对于一般的操作系统来说,数百个文件似乎并不过分,但我建议您事先阅读所有信息(可能一次打开多个文件,但考虑到调用“ open”失败的可能性,在这种情况下,您应该在关闭某些先前打开的文件后重试),然后进行处理并将结果存储在某些内部数据结构中。 Finally you can write the results back to the files, again in a parallel fashion but, again, being prepared to a failed attempt to open a file. 最终,您可以再次将结果以并行方式写回到文件中,但是同样要为打开文件的尝试失败做好准备。
2^(8*sizeof(filehandle))
distinct files. 在任何操作系统上,您最多只能打开2^(8*sizeof(filehandle))
不同的文件。 filehandle is a type used to access file contents. filehandle是一种用于访问文件内容的类型。 HANDLE, FILE*, int, etc. Depends on operating system. HANDLE,FILE *,int等。取决于操作系统。 You'll probably run out of memory before you reach this limit, though. 但是,在达到此限制之前,您可能会耗尽内存。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.