Fastest way to write huge matrices on disk with Java

Question

I need to write huge matrices to disk. I will have around one hundred matrices of 500.000 rows and 30 columns of double values. The first column of this matrix will work as an ID.

Then i want to have an hash map or similar that for a given ID returns the pointer to the line in which that id is in my matrix on disk.

To be clear I want to have an hash map that tell me in which position of the matrix is the line relative to that ID.

I don't want to write on a text file but use directly the memory with pointers to a particular position. It's like an hashmap but that will not be deleted at the end of the program.

I'm searching for the most efficient ways you think of doing this for read/write operations.

Answer 1

I suggest using a RandomAccessFile and writing your own code for the rest. You can write pointers in the file to other parts of the file using your own code in combination with the provided methods.