读取txt文件并使用c ++和boost内存映射文件快速解析它们

Question

Important edit : The problem is not what i stated, after manually profiling i understood that when i replace the line : "file >> x >> y >> z;" with the line "file.readline(buffer, size);"

it takes only 0.4 seconds, so the question is entirely different, how to parse the floats from the line, file>>x>>y>>z;

(i don't know if i should delete the question or not, because the original question is not relevant)

=== OLD === After vast research on the internet and stack overflow, i understood that the best way to read large files with c++ is by using memory mapped files.

I have a txt file, 15MB that on each line has 3 float separated by spaces.

I had this code :

ifstream file(path)
float x,y,z;
while(!file.eof())
  file >> x >> y >> z;

Which could read this file in 9.5 seconds.

In order to read the file faster using stackoverflow users i came up with this code, that if i understand it correctly uses memory mapped files and should read it faster Stream types in C++, how to read from IstringStream?

#include <iostream>
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/device/mapped_file.hpp>
namespace io = boost::iostreams;

int main()
{
    io::stream<io::mapped_file_source> str("test.txt");
    // you can read from str like from any stream, str >> x >> y >> z
    for(float x,y,z; str >> x >> y >> z; )
        std::cout << "Reading from file: " << x << " " << y << " " << z << '\n';
}

Unfortunately the speed remains the same, still 9.5 seconds.

Any suggestions ? Thanks

Answer 1

Streams are slow. Part is because the constraints that apply to them are onerous, part is because implementations have a tendency of being poorly optimized.

Try using Boost.Spirit parsers. While the syntax takes a bit of getting used to and compilation can sometimes be very slow, the runtime performance of Spirit is very high.