在不立即将整个文件加载到 memory 的情况下，分块读取大文件的最有效方法是什么？

Question

What is the most efficient general purpose way of reading "large" files (which may be text or binary), without going into unsafe territory? I was surprised how few relevant results there were when I did a web search for "rust read large file in chunks".

For example, one of my use cases is to calculate an MD5 checksum for a file using rust-crypto (the Md5 module allows you to add &[u8] chunks iteratively).

Here is what I have, which seems to perform slightly better than some other methods like read_to_end :

use std::{
    fs::File,
    io::{self, BufRead, BufReader},
};

fn main() -> io::Result<()> {
    const CAP: usize = 1024 * 128;
    let file = File::open("my.file")?;
    let mut reader = BufReader::with_capacity(CAP, file);

    loop {
        let length = {
            let buffer = reader.fill_buf()?;
            // do stuff with buffer here
            buffer.len()
        };
        if length == 0 {
            break;
        }
        reader.consume(length);
    }

    Ok(())
}

Answer 1

I don't think you can write code more efficient than that. fill_buf on a BufReader over a File is basically just a straight call to read(2) .

That said, BufReader isn't really a useful abstraction when you use it like that; it would probably be less awkward to just call file.read(&mut buf) directly.

Answer 2

I did it this way, I don't know if it is wrong but it worked perfectly for me, still don't know if it is the correct way tho..

use std::io;
use std::io::prelude::*;
use std::fs::File;

fn main() -> io::Result<()> 
{
    const FNAME: &str = "LargeFile.txt";
    const CHUNK_SIZE: usize = 1024; // bytes read by every loop iteration.
    let mut limit: usize = (1024 * 1024) * 15; // How much should be actually read from the file..
    let mut f = File::open(FNAME)?;
    let mut buffer = [0; CHUNK_SIZE]; // buffer to contain the bytes.

    // read up to 15mb as the limit suggests..
    loop {
        if limit > 0 {
            // Not finished reading, you can parse or process data.
            let _n = f.read(&mut buffer[..])?;

            for bytes_index in 0..buffer.len() {
               print!("{}", buffer[bytes_index] as char);
            }
            limit -= CHUNK_SIZE;
        } else {
            // Finished reading..
            break;
        }
    }    
    Ok(())
}