What is more efficient, reading word by word from file or reading a line at a time and splitting the string using C ?

Question

I want to develop an application in C where I need to check word by word from a file on disk. I've been told that reading a line from file and then splitting it into words is more efficient as less file accesses are required. Is it true?

Answer 1

If you know you're going to need the entire file, you may as well be reading it in as large chunks as you can (at the extreme end, you'll memory map the entire file into memory in one go). You are right that this is because less file accesses are needed.

But if your program is not slow, then write it in the way that makes it the fastest and most bug free for you to develop. Early optimization is a grievous sin.

Answer 2

Not really true, assuming you're going to be using scanf() and your definition of 'word' matches what scanf() treats as a word.

The standard I/O library will buffer the actual disk reads, and reading a line or a word will have essentially the same I/O cost in terms of disk accesses. If you were to read big chunks of a file using fread() , you might get some benefit — at a cost in complexity.

But for reading words, it's likely that scanf() and a protective string format specifier such as %99s if your array is char word[100]; would work fine and is probably simpler to code.

If your definition of word is more complex than the definition supported by scanf() , then reading lines and splitting is probably easier.

Answer 3

As far as splitting is concerned there is no difference with respect to performance. You are splitting using whitespace in one case and newline in another.

However it would impact in case of word in a way that you would need to allocate buffers M times, while in case of lines it will be N times, where M>N. So if you are adopting word split approach, try to calculate total memory need first, allocate that much chunk (so you don't end up with fragmented M chunks), and later get M buffers from that chunk. Note that same approach can be applied in lines split but the difference will be less visible.

Answer 4

This is correct, you should read them in to a buffer, and then split into whatever you define as 'words'. The only case where this would not be true is if you can get fscanf() to correctly grab out what you consider to be words (doubtful).

Answer 5

The major performance bottlenecks will likely be:

• Any call to a stdio file I/O function. The less calls, the less overhead.
• Dynamic memory allocation. Should be done as scarcely as possible. Ultimately, a lot of calls to malloc will cause heap segmentation.

So what it boils down to is a classic programming consideration: you can get either quick execution time or you can get low memory usage. You can't get both, but you can find some suitable middle-ground that is most effective both in terms of execution time and memory consumption.

To one extreme, the fastest possible execution can be obtained by reading the whole file as one big chunk and upload it to dynamic memory. Or to the other extreme, you can read it byte by byte and evaluate it as you read, which might make the program slower but will not use dynamic memory at all.

You will need a fundamental knowledge of various CPU-specific and OS-specific features to optimize the code most effectively. Issues like alignment, cache memory layout, the effectiveness of the underlying API function calls etc etc will all matter.

Why not try a few different ways and benchmark them?

Answer 6

Not actually answer to your exact question (words vs lines), but if you need all words in memory at the same time anyway, then the most efficient approach is this:

1. determine file size
2. allocate buffer for entire file plus one byte
3. read entire file to the buffer, and put '\\0' to the extra byte.
4. make a pass over it and count how many words it has
5. allocate char* (pointers to words) or int (indexes to buffer) index array, with size matching word count
6. make 2nd pass over buffer, and store addresses or indexes to the first letters of words to the index array, and overwrite other bytes in buffer with '\\0' (end of string marker).

If you have plenty of memory, then it's probably slightly faster to just assume the worst case for number of words: (filesize+1) / 2 (one letter words with one space in between, with odd number of bytes in file). Also taking the Java ArrayList or Qt QVector approach with the index array, and using realloc() to double it's size when word count exceeds current capacity, will be quite efficient (due to doubling=exponential growth, reallocation will not happen many times).