fwrite more than 4GB in Visual Studio

Question

I am trying to setup my visual studio so that I can fwrite 8GB at once.

I can see malloc allocating 8GB by keeping track of the memory using the monitor.

However, the return value of fwrite is zero and the output file size is only 4GB.

size_t s = fwrite(result, sizeof(unsigned int), 0x80000000, fout);

I am using x64, Release mode.

Is there any other settings I should use?

Answer 1

There is not a lot of point in using the oldest C runtime functions for working with large data, as both the Windows API and C++ have better ways of handling larger data, such as memory mapped files. For C++ memory maps files, boost has a couple of implementations.

If you really want to use fwrite, then try splitting it up, as Visual C++'s implementation of fwrite will not write large arrays in one go:

fwrite is implemented on top of the Windows WriteFile function, which only can write a DWORD worth of bytes, furthermore see this comment from its implementation, found in C:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\VC\\crt\\src\\fwrite.c .

/***
*size_t fwrite(void *buffer, size_t size, size_t count, FILE *stream) -
*       write to the specified stream from the specified buffer.
*
*Purpose:
*       Write 'count' items of size 'size' to the specified stream from
*       the specified buffer. Return when 'count' items have been written
*       or no more items can be written to the stream.
*
*Entry:
*       buffer  - pointer to user's buffer
*       size    - size of the item to write
*       count   - number of items to write
*       stream  - stream to write to
*
*Exit:
*       Returns the number of (whole) items that were written to the stream.
*       This may be less than 'count' if an error or eof occurred. In this
*       case, ferror() or feof() should be used to distinguish between the
*       two conditions.
*
*Notes:
*       fwrite will attempt to buffer the stream (side effect of the _flsbuf
*       call) if necessary.
*
*       No more than 0xFFFE bytes may be written out at a time by a call to
*       write(). Further, write() does not handle huge buffers. Therefore,
*       in large data models, the write request is broken down into chunks
*       that do not violate these considerations. Each of these chunks is
*       processed much like an fwrite() call in a small data model (by a
*       call to _nfwrite()).
*...

Note the section beginning No more than 0xFFFE bytes may be written out at a time by a call to write() - apart from simplicity, there isn't an advantage calling it with a large chunk of memory vs making several calls with smaller chunks and checking the return.

Answer 2

I'm not sure why you're getting a zero return value. Per the MS documentation for fwrite() :

Syntax

 size_t fwrite( const void *buffer, size_t size, size_t count, FILE *stream ); 

you've written more than zero items in order to get a 4 GB output file. Exactly how do you know the return value is zero?

Second, there's no point in trying to write such a huge amount in one line of code - partial writes are always possible, where a write attempt of any size can result in less than that amount being written. And when you write gigabytes at a time, partial writes are going to be very likely, so you have to write code to handle the partial writes anyway. Performance-wise, even on extremely high-speed disk systems you'll stop seeing performance improvements once write sizes get into the megabyte range. On consumer-grade commodity systems, performance won't likely improve once your write requests get to a few kilobytes.

Third, if you're not adjusting the buffer size of your FILE * object, it's probably buffered in chunks of 4 or 8 KB anyway. It's not a coincidence that's about the size where larger chunks don't result in much if any performance improvement on the majority of disk systems.

Finally, as @Andriy Berestovskyy stated in his answer, you might be running into a file system limitation. See if you can write a file larger than 4GB using multiple fwrite() calls.