如何从大文本文件读取到数组C ++

Question

I'm trying to read from a very large text file with two columns, it's a web graph, something like this: (except it has 40 million rows).

1 2

1 3

2 1 ...

so i wanted to read from the txt file to myArray[mysize][2], and i used the code:

ifstream file("web-graph.txt");
if(file.is_open())
{
    for(int i = 0; i < mysize; i++)
    {
        file >> myArray[i][0];          
        file >> myArray[i][1];
    }
}

the problem is that it takes a long time to read such a big file. so is there any other way to read from the file that doesn't take this much time?

Answer 1

Yes, possibly, subject to profiling, but you won't like the answer. If you make the file smaller it might be quicker to read. How? Save it as binary, rather than text. Be aware this will stop you being able to use the nice high level streaming operators. You will have to use lower level things instead, which might give you even more speedup.
It might be better if you ask yourself why you are reading the whole file into memory. Again, if you made the file binary you could seek to the specific lines you are after.
If you are performing a calculation on the file, perhaps you can process it as you go, or in chunks.

Answer 2

Yes, you're definitely doing it the slow (but pretty) way. You have 2 options to be faster:

if( you have enough memory ) { Read the entire file into memory and, and then parse the file }

else { Read large chunks of the file at a time into memory, and then parse the file }

Either way, the loading looks something like this...

std::ifstream is(filename);
is.seekg(0, std::ios::end);
auto length = is.tellg();

std::string buffer;

if(length > 0)
{
    buffer.resize(static_cast<std::string::size_type>(length));
    is.seekg(0);
    is.read(&buffer.front(), length);
}

And then you would put it in a stringstream...

std::stringstream ss(buffer);

and parse it, potentially exactly how you were doing it before...

for(int i = 0; i < mysize; i++)
{
    ss >> myArray[i][0];          
    ss >> myArray[i][1];
}