大Numpy Arrays的高效串联

Question

I'm running a process that creates a very large number of feature vectors (as numpy arrays) and stacks them into a single array. This process is currently very memory intensive and I'm looking for a more memory efficient way to run it.

Currently I generate feature vectors in batches of 100000 and concatenate them together.

all_features = None

for i in range(0, num_entries, 100000):
    features = get_features(entries[i:i+100000]) # generate a batch of 100,000 feature vectors
    features = np.array(features)

    if all_features is not None:
        all_features = np.concatenate([all_features, features])
    else:
        all_features = features

    del features
    gc.collect()

I've found that iteratively concatenating feature vectors, then deleting the intermediate features object is more memory efficient than generating all features at once and concatenating them all at once. I believe this is because np.concatenate allocates a new object in memory. (Trying to generate all feature vectors at once, then concatenating blows up memory).

That said, it gets to a point where running the concatenation near the end of the loop still requires about 30 GB of memory (which is immediately freed after the concatenation is run).

Basically I have enough memory on my instance to store the full feature set, but the memory jumps from packing things into a single array make me run out of memory.

Is there a more memory efficient way of running this?

Answer 1

If total size of all_features is known, I'd suggest to allocate it in advance all_features=np.zeros(...) and then populate it in the loop. Thus you get rid of multiple reallocations, deletions and np.concatenate() calls.

Answer 2

Make your get_features function a generator and then use np.fromiter to create the array.

Simple Example

def gen_values():
    for i in range(1000000): 
        yield i

a = np.fromiter(gen_values(), dtype=int)

You need to specify the dtype in np.fromiter and you can optionaly specify the number of elements to get from the generator with count . While it is optional, it is much better to specify count such that numpy can pre-allocate the output array, instead of resizing it on demand.