c++读取文件太慢

Question

I'm trying to to read ~36KB and it would take ~20 seconds to finish this loop:

ifstream input_file;

input_file.open("text.txt");
if( !(input_file.is_open()) )
{
    cout<<"File not found";
    exit(1);
}

std::string line;
stringstream line_stream;   //to use << operator to get words from lines

int lineNum=1;

while( getline(input_file,line) )   //Read file line by line until file ends
{
    line_stream.clear();    //clear stream
    line_stream << line;    //read line
    while(line_stream >> word)  //Read the line word by word until the line ends
    {
        //insert word into a linked list...
    }
    lineNum++;
}
input_file.close();

Any help would be appreciated.

Answer 1

stringstream::clear() does not clear all context inside it. It only resets the error and EOF flags, see http://en.cppreference.com/w/cpp/io/basic_ios/clear .

The result is your line_stream accumulates all previous lines and the inner loop will run words over all the accumulated lines again and again.

So the total time you spend is about O(n^2) compared to O(n) of what you expect it to be.

Instead of using the same object across each line, you could define the new line_stream instance inside the while loop to have a brand new and also empty one. Like this:

fstream input_file;

input_file.open("text.txt");
if( !(input_file.is_open()) )
{
    cout<<"File not found";
    exit(1);
}

std::string line;

int lineNum=1;

while( getline(input_file,line) )   //Read file line by line until file ends
{
    stringstream line_stream;   // new instance, empty line.
    line_stream << line;    //read line
    while(line_stream >> word)  //Read the line word by word until the line ends
    {
        //insert word into a linked list...
    }
    lineNum++;
}
input_file.close();

Answer 2

You could attempt the following:

std::ifstream file("text.txt");
std::string str;

while (std::getline(file, str))
{
    cout << str; //call function to to retrieve words of str in memory not in file 
}

I ran your code in 11ms, but with the mentioned option in 8ms. May be it works for you.

Answer 3

Try compiling with build flag -O2 or -O3 .

I was surprised to see that a simple for-loop to read a 1GB file took 4.7 seconds, whereas another higher level language (Dart) did it in 3.x seconds.

After enabling this flag, runtime dropped to 2.1 seconds.