How to read/write a specific number of bytes to file

Question

I am looking to create a file by structuring it in size blocks. Essentially I am looking to create a rudimentary file system.

I need to write a header, and then an "infinite" possible number of entries of the same size/structure. The important parts are:

• Each block of data needs to be read/writable individually
• Header needs to be readable/writable as its own entity
• Need a way to store this data and be able to determine its location in the file quickly

The would imagine the file would resemble something like:

[HEADER][DATA1][DATA2][DATA3][...]

What is the proper way to handle something like this? Lets say I want to read DATA3 from the file, how do I know where that data chunk starts?

Answer 1

If I understand you correctly and you need a way to assign a kind of names/IDs to your DATA chunks, you can try to introduce yet another type of chunk.

Let's call it TOC (table of contents). So, the file structure will look like [HEADER][TOC1][DATA1][DATA2][DATA3][TOC2][...] .

TOC chunk will contain names/IDs and references to multiple DATA chunks. Also, it will contain some internal data such as pointer to the next TOC chunk (so, you might consider each TOC chunk as a linked-list node).

At runtime all TOC chunks could be represented as a kind of HashMap , where key is a name/ID of the DATA chunk and value is its location in the file.

Answer 2

We can store in the header the size of chunk. If the size of chunks are variable, you can store pointers which points to actual chunk. An interesting design for variable size is in postgres heap file page. http://doxygen.postgresql.org/bufpage_8h_source.html

Answer 3

I am working in reverse but this may help.

I write decompilers for binary files. Generally there is a fixed header of a known number of bytes. This contains specific file identification so we can recognize the file type we are dealing with.

Following that will be a fixed number of bytes containing the number of sections (groups of data) This number then tells us how many data pointers there will be. Each data pointer may be four bytes (or whatever you need) representing the start of the data block. From this we can work out the size of each block. The decompiler then reads the blocks one at a time to get the size and location in the file of each data block. The job then is to extract that block of bytes and do whatever is needed.

We step through the file one block at a time. The size of the last block is the start pointer to the end of the file.