如何避免内存错误

Question

How should I proceed in order to be able to plot some data coming out from a big pytable (17GB).

If I try to store the values that I need, I get Memory Error , something like this:

for row in tab.iterrows():

    x.append(row['date'])
    y.append(row['temperature'])

    #this will not end, raises Memory Error

If I try to live update a plot with the data, without storing any values, I get this message:

MemoryError
QImage: out of memory, returning null image

So then? What should I do If I need to plot some data from such a big table?

Note:

Working with python 2.7 32 bits on a Windows 64-bit machine 8GB

I know that a solution would be to use python 64, but, it should be possible to handle in python 32 too.

Answer 1

Try to find out what section of your data you really want to plot. It doesn't make sense to have > 1 million dots on your screen. You say it yourself: ... if I need to plot __some__ data from such a big table .

I haven't worked with PyTables yet, but the documentation says it's returning numpy objects, so no need to iterate over the rows.

histogram, xedges, yedges = 
   numpy.histogram2d(
      tab.col('date'),
      tab.col('temperature'),
      bins=50)

pyplot.imshow(heatmap)
pyplot.show()