Why does BufReader::read not always completely fill the given buffer?

Question

I expected this code to read 3 bytes on each loop and not print, yet every 8000 or so bytes of the file, it only reads two bytes:

use std::error::Error;
use std::fs::File;
use std::io::prelude::*;
use std::io::BufReader;
use std::io::BufWriter;
use std::io::SeekFrom;
use std::path::Path;

fn main() -> std::io::Result<()> {
    let sequence: [u8; 3] = [1, 2, 3];
    let file_path = Path::new("./test_file");
    //fill a file with 1,2,3,1,2,3...
    {
        let mut output_file: std::io::BufWriter<std::fs::File>;
        output_file = BufWriter::new(File::create(file_path)?);
        for _i in 0..100000 {
            match output_file.write(&sequence) {
                Err(why) => panic!("could not write {}", Error::description(&why)),
                Ok(_) => {}
            }
        }
    }
    //read the file 3 bytes at a time
    {
        let mut input_file: std::io::BufReader<std::fs::File>;
        input_file = BufReader::new(File::open(file_path)?);
        for i in 0..100000 {
            let mut raw = [0; 3];
            let result = match input_file.read(&mut raw) {
                Err(why) => panic!("could not read {}", Error::description(&why)),
                Ok(x) => x,
            };
            // print if something other than 3 bytes were read
            if result != 3 {
                println!(
                    "file pos {}, data read {}, buffer = [{},{},{}]",
                    i * 3,
                    result,
                    raw[0],
                    raw[1],
                    raw[2]
                );
            }
        }
    }
    Ok(())
}

Compiled on a Mac with rustc problem.rs and run with ./problem .

Output:

file pos 8190, data read 2, buffer = [1,2,0]
file pos 16383, data read 2, buffer = [3,1,0]
file pos 24576, data read 2, buffer = [2,3,0]
file pos 32769, data read 2, buffer = [1,2,0]
file pos 40962, data read 2, buffer = [3,1,0]
file pos 49155, data read 2, buffer = [2,3,0]
file pos 57348, data read 2, buffer = [1,2,0]
file pos 65541, data read 2, buffer = [3,1,0]
...

This seems to suggest something to do with an internal 8192 sized buffer.

Why don't I get 3 bytes each time? I get similar results reading 5 bytes at a time.

Answer 1

The default buffer size is 8KB. See the docs for BufReader::new .

8192 bytes isn't divisible by 3, so you get a couple of trailing bytes at the end of each buffer.

You can set the buffer size to something other than 8KB by using the with_capacity constructor .