How to read a big (3-4GB) file that doesn't have newlines into a numpy array?

Question

I have a 3.3gb file containing one long line. The values in the file are comma separated and either floats or ints. Most of the values are 10 . I want to read the data into a numpy array. Currently, I'm using numpy.fromfile :

>>> import numpy
>>> f = open('distance_matrix.tmp')
>>> distance_matrix = numpy.fromfile(f, sep=',')

but that has been running for over an hour now and it's currently using ~1 Gig memory, so I don't think it's halfway yet.

Is there a faster way to read in large data that is on a single line?

Answer 1

This should probably be a comment... but I don't have enough reputation to put comments in.

I've used hdf files, via h5py, of sizes well over 200 gigs with very little processing time, on the order of a minute or two, for file accesses. In addition the hdf libraries support mpi and concurrent access.

This means that, assuming you can format your original one line file, as an appropriately hierarchic hdf file (eg make a group for every `large' segment of data) you can use the inbuilt capabilities of hdf to make use of multiple core processing of your data exploiting mpi to pass what ever data you need between the cores.

You need to be careful with your code and understand how mpi works in conjunction with hdf, but it'll speed things up no end.

Of course all of this depends on putting the data into an hdf file in a way that allows you to take advantage of mpi... so maybe not the most practical suggestion.

Answer 2

Consider dumping the data using some binary format. See something like http://docs.scipy.org/doc/numpy/reference/generated/numpy.save.html This way it will be much faster because you don't need to parse the values.

If you can't change the file type (not the result of one of your programs) then there's not much you can do about it. Make sure your machine has lots of ram (at least 8GB) so that it doesn't need to use the swap at all. Defragmenting the harddrive might help as well, or using a SSD drive.

An intermediate solution might be a C++ binary to do the parsing and then dump it in a binary format. I don't have any links for examples on this one.