fastest way to parse/jump through large binary file with sized elements

Question

I need to parse binary files which contain a sequence of elements. The format of an element is as follows: 4 bytes: name of element 4 bytes: size of the element variable size: data for the element

I just need to parse through the file and extract the name, position and size of each element. Typical element size is around 100kb, and typical file size is around 10GB.

What is the fastest way of going through such a file? Read all of the file's data, seek to the next element, other approach?

Does it make a difference if the file is local or over the network?

Answer 1

The one thing you do not want to do is to use unbuffered reads (ie OS calls) to read every individual element. You can get an OK performance by the naive approach of buffered reads. If memory is not a concern whatsoever, you might squeeze some time by using memory-mapped files, and having a pre-fetcher thread to populate the mapping.