C++ ifstream.getline() significantly slower than Java's BufferedReader.readLine()?

Question

I'm in the process of rewriting one of my Android applications to take advantage of the NDK and one of the first things it has to do every time is open a 1.5MB text file (approximately 150k lines) and put every line in a data structure. When I did this operation using Java's BufferedReader.readLine(), reading the file from the SD card takes ~2.5 seconds. Here's the code I used for this:

try {
    BufferedReader br = new BufferedReader(new FileReader("/sdcard/testfile.txt"));
    String thisLine;
    while ((thisLine = br.readLine()) != null) {
        Log.d(TAG, thisLine);
    }
} catch (IOException e) {
    //Log error
}

Using C++ with ifstream takes MUCH longer...around 3 minutes for the same file. Here's the code I used in C++:

char buffer[256];
ifstream ifs;
ifs.open("/sdcard/testfile.txt", ifstream::in);
if (ifs.is_open()) {
    while (!ifs.eof()) {
        ifs.getline (buffer,100);
        LOGD(buffer);
    }
}

I'm pretty rusty on C++, but can't think of any logical explanation for the increased read time. For a while I thought it might be the LOGD function, but I tried taking that out altogether and the read time wasn't really helped much at all. Does anyone have any ideas on what the issue could be? Is there any faster way to read a file line by line in C++? Thanks.

Answer 1

One thought is the stdio synchronization might be slowing you down. That can be turned off. I don't know if that would account for all of the difference, but you could try. Also, you're not using eof() correctly. Finally, I'd use the std::string version of getline()

std::ios::sync_with_stdio(false);
ifstream ifs("/sdcard/testfile.txt");
std::string line;
while (getline(ifs, line))
{
    LOGD(line);
}

I haven't tested this code, but you can try it and see if it makes a difference.

Answer 2

Is it possible that the stream is unbuffered, and it's doing an SD access for each byte of data? To provide a buffer, do the following (size as you feel appropriate).

ifstream ifs;
char stream_buffer[4096];
ifs.rdbuf()->pubsetbuf(stream_buffer, sizeof(stream_buffer) );
ifs.open(argv[1]);

Answer 3

C++ does not buffer streams for you (edit: they won't by default , see Dave Smith's solution). I will tell you that your code will be slow on a normal platter based disk. I don't have a lot of experience with android, fwiw.

I generally use something like this:

struct buffered_reader {
    buffered_reader(std::istream &data_) : data(data_), done(false) {}
    bool next(std::string &line) {
        if (!lines.size()) {
            if (done)
                return false;
            std::string line;
            for (size_t i = 0; i < 500; i++) {
                std::getline(data, line);
                if (data.eof()) {
                    done = true;
                    break;
                }
                lines.push_back(line);
            }
        }
        line = lines.front();
        lines.pop_front();
        return true;
    }
    std::istream &data;
    bool done;

    std::deque<std::string> lines;
};

TEST(blah) {
    std::stringstream ss;
    ss << "a" << std::endl;
    ss << "a" << std::endl;
    ss << "a" << std::endl;
    ss << "a" << std::endl;

    buffered_reader reader(ss);
    std::string line;
    while(reader.next(line)) {
        std::cout << line << std::endl;
    }
}

This isn't in production anywhere, so no warranties beyond the testing you see here ;)