How to build an array when you don't know the final size?

Question

I want to read in files containing points of x,y,z data into an array one at a time. So the first file might be 20 points, then next one could be 45 points, etc. The final size of the array would be (nPoints, 3). However, I don't know the total numbers of points. How do you declare the final array before you read in the data?

What I have seen is best practice is to declare the final array as np.empty(nPoints, 3) and then insert the data using indexing.

What I have done is just create a (1,3) array of zeros, use np.vstack to add the data, and then just deleted the first row. But that seems like naive programming to me.

Answer 1

Here you are executing a sequential operation (reading your file) and not doing vectorized computation. Therefore, there is no true improvement to declare before-hand np.array .

You should just use regular python list, so you can append elements as you read the file. Once all elements are added, you can then convert it to np.array .

Well, if you are super strict on memory usage, you might prefer compute the number of elements first by counting them when reading the file, to then initialize a np.array of the right size and fill by reading the file again. This is because lists are a bit bigger than their length (they basically store additional memory addresses in order to easily append elements), unlike np.array who has a fixed size.