How can I read successive arrays from a binary file using `np.fromfile`?

Question

I want to read a binary file in Python, the exact layout of which is stored in the binary file itself.

The file contains a sequence of two-dimensional arrays, with the row and column dimensions of each array stored as a pair of integers preceding its contents. I want to successively read all of the arrays contained within the file.

I know this can be done with f = open("myfile", "rb") and f.read(numberofbytes) , but this is quite clumsy because I would then need to convert the output into meaningful data structures. I would like to use numpy's np.fromfile with a custom dtype , but have not found a way to read part of the file, leaving it open, and then continue reading with a modified dtype .

I know I can use os to f.seek(numberofbytes, os.SEEK_SET) and np.fromfile multiple times, but this would mean a lot of unnecessary jumping around in the file.

In short, I want MATLAB's fread (or at least something like C++ ifstream read ).

What is the best way to do this?

Answer 1

You can pass an open file object to np.fromfile , read the dimensions of the first array, then read the array contents (again using np.fromfile ), and repeat the process for additional arrays within the same file.

For example:

import numpy as np
import os

def iter_arrays(fname, array_ndim=2, dim_dtype=np.int, array_dtype=np.double):

    with open(fname, 'rb') as f:
        fsize = os.fstat(f.fileno()).st_size

        # while we haven't yet reached the end of the file...
        while f.tell() < fsize:

            # get the dimensions for this array
            dims = np.fromfile(f, dim_dtype, array_ndim)

            # get the array contents
            yield np.fromfile(f, array_dtype, np.prod(dims)).reshape(dims)

Example usage:

# write some random arrays to an example binary file
x = np.random.randn(100, 200)
y = np.random.randn(300, 400)

with open('/tmp/testbin', 'wb') as f:
    np.array(x.shape).tofile(f)
    x.tofile(f)
    np.array(y.shape).tofile(f)
    y.tofile(f)

# read the contents back
x1, y1 = iter_arrays('/tmp/testbin')

# check that they match the input arrays
assert np.allclose(x, x1) and np.allclose(y, y1)

If the arrays are large, you might consider using np.memmap with the offset= parameter in place of np.fromfile to get the contents of the arrays as memory-maps rather than loading them into RAM.