C ++逐步读取大文件

Question

I've been having a problem that I not been able to solve as of yet. This problem is related to reading files, I've looked at threads even on this website and they do not seem to solve the problem. That problem is reading files that are larger than a computers system memory. Simply when I asked this question a while ago I was referred too using the following code.

    string data("");
getline(cin,data);
std::ifstream is (data);//, std::ifstream::binary);
if (is) 
{
    // get length of file:
    is.seekg (0, is.end);
    int length = is.tellg();
    is.seekg (0, is.beg);
    // allocate memory:
    char * buffer = new char [length];
    // read data as a block:
    is.read (buffer,length);
    is.close();
    // print content:

    std::cout.write (buffer,length);
    delete[] buffer;
}
system("pause");

This code works well apart from the fact that it eats memory like fat kid in a candy store. So after a lot of ghetto and unrefined programing, I was able to figure out a way to sort of fix the problem. However I more or less traded one problem for another in my quest.

#include <iostream>
#include <vector>
#include <string>
#include <fstream>
#include <stdio.h> 
#include <stdlib.h>
#include <iomanip>
#include <windows.h>
#include <cstdlib>
#include <thread>

using namespace std;
/*======================================================*/
    string *fileName = new string("tldr");
    char data[36];
    int filePos(0); // The pos of the file
    int tmSize(0); // The total size of the file    

    int split(32);
    char buff;
    int DNum(0);
/*======================================================*/



int getFileSize(std::string filename) // path to file
{
    FILE *p_file = NULL;
    p_file = fopen(filename.c_str(),"rb");
    fseek(p_file,0,SEEK_END);
    int size = ftell(p_file);
    fclose(p_file);
    return size;
}

void fs()
{
    tmSize = getFileSize(*fileName);
    int AX(0);
    ifstream fileIn;
    fileIn.open(*fileName, ios::in | ios::binary);
    int n1,n2,n3;
    n1 = tmSize / 32;

    // Does the processing
    while(filePos != tmSize)
    {
        fileIn.seekg(filePos,ios_base::beg);
        buff = fileIn.get();
        // To take into account small files
        if(tmSize < 32)
        {
            int Count(0);
            char MT[40];
            if(Count != tmSize)
            {
                MT[Count] = buff;
                cout << MT[Count];// << endl;
                Count++;
            }
        }
        // Anything larger than 32
        else
        {
            if(AX != split)
            {
                data[AX] = buff;
                AX++;
                if(AX == split)
                {

                    AX = 0;
                }
            }

        }
        filePos++;
    }
    int tz(0);
    filePos = filePos - 12;

    while(tz != 2)
    {
        fileIn.seekg(filePos,ios_base::beg);
        buff = fileIn.get();
        data[tz] = buff;
        tz++;
        filePos++;
    }

    fileIn.close();
}

void main ()
{
    fs();
    cout << tmSize << endl;
    system("pause");
}

What I tried to do with this code is too work around the memory issue. Rather than allocating memory for a large file that simply does not exist on a my system, I tried to use the memory I had instead which is about 8gb, but I only wanted to use maybe a few Kilobytes of it if at all possible. To give you a layout of what I am talking about I am going to write a line of text. "Hello my name is cake please give me cake" Basically what I did was read said piece of text letter by letter. Then I put those letters into a box that could store 32 of them, from there I could use something like xor and then write them onto another file. The idea in a way works but it is horribly slow and leaves off parts of files. So basically how can I make something like this work without going slow or cutting off files. I would love to see how xor works with very large files. So if anyone has a better idea than what I have, then I would be very grateful for the help.

Answer 1

To read and process the file piece-by-piece, you can use the following snippet:

// Buffer size 1 Megabyte (or any number you like)
size_t buffer_size = 1<<20;
char *buffer = new char[buffer_size];

std::ifstream fin("input.dat");

while (fin)
{
    // Try to read next chunk of data
    fin.read(buffer, buffer_size);
    // Get the number of bytes actually read
    size_t count = fin.gcount();
    // If nothing has been read, break
    if (!count) 
        break;
    // Do whatever you need with first count bytes in the buffer
    // ...
}

delete[] buffer;

The buffer size of 32 bytes, as you are using, is definitely too small. You make too many calls to library functions (and the library, in turn, makes calls (although probably not every time) to OS, which are typically slow, since they cause context-switching). There is also no need of tell/seek.

Answer 2

If you don't need all the file content simultaneously, reduce the working set first - like a set of about 32 words, but since XOR can be applied sequentially, you may further simplify the working set with constant size, like 4 kilo-bytes.

Now, you have the option to use file reader is.read() in a loop and process a small set of data each iteration, or use memmap() to map the file content as memory pointer which you can perform both read and write operations.