Keras - Generator for large dataset of Images and Masks

Question

I'm trying to build a Model that has Images for both its Inputs and Outputs (masks). Because of the size of the Dataset and my limited Memory, I tried using the Generator Approach introduced in the Keras Documentation :

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

model.fit_generator(
    train_generator,
    samples_per_epoch=2000,
    nb_epoch=50)

Everything seems to work except when the code gets to this line:

train_generator = zip(image_generator, mask_generator)

it seems the process of zipping the two lists explicitly makes them generate their content and the system starts consuming lots of RAM until it runs out of Memory.

The point of using Generators is to avoid running out of RAM while this piece of code is exactly doing the opposite.

Is there any way to fix this problem?

Answer 1

You can user itertools.izip() to return an iterator instead of a list.

itertools.izip(*iterables)

Make an iterator that aggregates elements from each of the iterables. Like zip() except that it returns an iterator instead of a list. Used for lock-step iteration over several iterables at a time.