Is it better to store temp data in arrays or save it to file for access later?

Question

This is a broad question. I am running a very long simulation (in Python) that generates a sizeable amount of data (about 10,000 729*729 matrices). I only need the data to plot a couple of graphs and then I'm done with it. At the moment I save the data in (numpy) arrays. When the simulation is complete I plot the data.

One alternative would be to write the data to a file, and then access the file after simulation to plot graphs etc.

In general is there consensus on the best (ie quickest) way to manage large temporary data sets. Is either of these "best practice"?

Answer 1

Try to make the data obsolete as fast as possible by further processing/accumulating eg plotting immediately.

You did not give details about the memory/storage needed. for sparse matrices there are efficient representations. if your matrices are not sparse there are roughly 500k entries per matrix and therefore 5G entries altogether. without knowing your data type this could be typically 40GB of memory.

I strongly suggest to review your algorithms for achieving a smaller memory footprint.