how to write data to file larger than 2GB with fstream in c++

Question

I have searched a TON to find a way, using fstream or other methods, to write data into a large file, but have not found any answers.

My problem is, i'm downloading data from the internet and then writing data into pre-allocated file. Everything seems OK with a file smaller than 2GB, but when trying with a pre-allocated file larger than 2GB, it gives an error when moving the stream pointer to a specific position (eg last position of file) in the pre-allocated file to write data, it seems this cannot be done, tellp allway return -1 .

Here is my writing data method, because write to pre-allocate file, i'm not using FILE, or boost:iosstream. I'm using visual studio 2008 with .Net 3.5. Thanks for reading and helping me ^.^

size_t writeData(void *ptr, size_t size, size_t nmemb, Downloader *objDownloader){
    size_t written;     

    if (objDownloader->_stop == false && objDownloader->mystream.is_open()){

        objDownloader->mystream.seekp(objDownloader->_posSeek, ios::beg);           
        written=size * nmemb;
        objDownloader->mystream.write(static_cast<char*>(ptr), written);

        objDownloader->_posSeek += size*nmemb;
        objDownloader->_downloaded += size*nmemb;

        objDownloader->mystream.seekp(objDownloader->_currentPosition, ios::beg);   
        objDownloader->mystream.write((char*)&objDownloader->_downloaded, sizeof(long long));

        long long  pos = objDownloader->mystream.tellp();           
        cout << "POS TEL: " << pos << endl;

        return size * nmemb;
    }
    else{

        if (objDownloader->mystream.is_open())
            objDownloader->mystream.close();            
        return -1;
    }

}

Answer 1

Your problem is not related to fstreams but to process memory.

The following code will generate a file of 3GB without any problem, use tellp() to show the progress, and positions with seekp() to the end of the file:

cout << "size_t is limted to " << numeric_limits<size_t>::max() << " bytes (" << numeric_limits<size_t>::max() /1024/1024<<" MB)"<< endl;
cout << "streamsize is limited to " << numeric_limits<streamsize>::max() << " bytes (" << numeric_limits<streamsize>::max() / 1024 / 1024/1024/104 << " TB)" << endl;

static char mbloc[1024*1024]; 
fill_n(mbloc, sizeof(mbloc), 'A'); 
const size_t blocs = 3*1024; 
ofstream os("test.out"); 
for(int i = 0; i < blocs && os; i++) {
    os.write(mbloc, sizeof(mbloc));
    if(i % 512)
        cout << "tellp() -> " << os.tellp() << "\r";
}
cout << endl;
os.seekp(0, ios::end); 
cout << os.tellp() << " end"<<endl;  
os.close();
cout << "File generation finished"; 

fstreams are in fact thought to be incredibly large. This is why ostream::write() uses a count of type streamsize .

The fact that tellp() returns -1 mens therefore that another failure took place. You could identify the origin of this failure by checking the status of your fstream operations, to identify which succeeds and which fails.

Your statement here shows that you try to write everything from a memory buffer:

     objDownloader->mystream.write(static_cast<char*>(ptr), written);

Unfortunately you will experience a problem with the limit of 2GB that applies to a 32 bit process under windows. This might cause a failure of writing, not because of fstream limits, but memory limits of the underlying OS.

If you compile for 64bit and run it on a 64 bit process, you could extend this limit to 8TB, if you have enough memory on your computer. But if you manipulate such huge data you should better write on disk in smaller chunks, without having to keep the full file in memory.