fread: How to know the size of the buffer?

Question

In C, when I am reading a file, or some other input, all at once with fread how would I know what size to declare the buffer with?

char buffer[1024];
fp = popen("ls /", "r");
fread(buffer, 1, sizeof(buffer), fp);

The input data could have 1000 lines or 1 line, or even 100000 lines or more.

Is there any general rule for it?

Answer 1

No, there's no general rule, because it depends entirely on what you plan to do with the data. Do you just need to parse it as it comes? That would be ideal. If you know that your input has useful data "samples" in chunks of X bytes, then simply read X bytes at a time and handle them as you go.

If you do need to copy the entire input into a buffer, then you'll have to take an initial guess, and allocate more memory if your guess is insufficient.

• In C++ you can just use std::vector (or std::deque if the data need not be contiguous in memory) to automatically expand the buffer as needed.

• In C you'll have to malloc first, then realloc inside your read loop when you run out of space in what you've already allocated.

  I suggest mimicking the behaviour of std::vector by making your buffer expand exponentially (multiplying by a factor of something like 1.5 or 2 each time), to help reduce the number of times you need to do this. So, say you first allocate 1,024 bytes. When that runs out, allocate 2,048. When that runs out, allocate 4,096. And so forth.

  Only you can decide what a good starting size is, based on your use case and expected nominal inputs.