用pytables构造巨大的numpy数组

Question

I generate feature vectors for examples from large amount of data, and I would like to store them incrementally while i am reading the data. The feature vectors are numpy arrays. I do not know the number of numpy arrays in advance, and I would like to store/retrieve them incrementally.

Looking at pytables, I found two options:

1. Arrays : They require predetermined size and I am not quite sure how much appending is computationally efficient.
2. Tables : The column types do not support list or arrays.

Answer 1

If it is a plain numpy array, you should probably use Extendable Arrays (EArray) http://pytables.github.io/usersguide/libref/homogenous_storage.html#the-earray-class

If you have a numpy structured array, you should use a Table.

Answer 2

Can't you just store them into an array? You have your code and it should be a loop that will grab things from the data to generate your examples and then it generates the example. create an array outside the loop and append your vector into the array for storage!

array = []
for row in file:
    #here is your code that creates the vector
    array.append(vector)

then after you have gone through the whole file, you have an array with all of your generated vectors! Hopefully that is what you need, you were a bit unclear...next time please provide some code.

Oh, and you did say you wanted pytables, but I don't think it's necessary, especially because of the limitations you mentioned