Reading and writing binary files in Qt

Question

I'm going to be working with binary files in a Qt project, and, being a little new to Qt, am not sure whether or not I should use a QVector<quint8> or a QByteArray to store the data. The files could be very small (< 1MiB) or very large (> 4GiB). The size is unknown until runtime.

I need to be able to have random seek and be able to process operations on every byte in the file. Would a memory mapped file be of any use here?

Thank you for any suggestions.

Answer 1

Loading whole large files in memory, be it QVector or QByteArray is probably not a good solution.

Assuming the files have some kind of structure, you should use QFile::seek to position yourself at the start of a "record" and use qint64 QIODevice::read ( char * data, qint64 maxSize ) to read one record at a time in a buffer of your choice.

Answer 2

QIODevice::write has an overload for QByteArray if that affects your decision. QDataStream may be worth looking at for larger data. At the end of the day it's really up to you as various containers would work.

Edit:

I think basic file I/O using whatever buffer you prefer is probably all you need. Use objects like QFile , QDataStream , QByteArray , etc. You could read and process only parts of the file with circular buffers to save memory especially if dealing with audio, video, or something that lends itself to streams. If there is a known structure to the file like XML, CSV, etc that also makes partial reads and processing easier as you can go line by line or tag by tag.

Memory mapped files use virtual memory to achieve faster I/O by basically making a copy of a file on disk in a virtual memory segment which is then capable of being used by the application as if it was just process memory. Being able to treat a file as process memory allows you to do in place editing which is faster than having to seek to a position from the beginning of a file and faster than making OS dependent API calls and dealing with hard disk read/writes. There does tend to be a fair amount of overhead to memory mapped files and there are some possible limitations depending on how paging is implemented in your target platform or what architecture you're using. In Qt you would have to design your own objects to use memory mapped files and historically I believe linux systems support this functionality better than windows.