Access in-memory data from multiple Jupiter notebooks

Question

I have multiple GPUs, to train different neural networks in parallel, but unfortunately due to the large size of the data I am limited, by the CPU memory and can't run multiple models on different notebooks because I would have to load in the data for each notebook.

Is there a way to load in the data from one notebook and access it through another?

I am using TensorFlow/Keras

Answer 1

Instead of trying to load everything into memory, use a generator function. This will read data into memory in batches, then push those batches through the neural network. Keras has built-in generator functions exactly for the purpose of reading in batches of image data from disk.

Please refer to https://keras.io/preprocessing/image/

Here is a small example script:

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

model.fit_generator(
        train_generator,
        steps_per_epoch=2000,
        epochs=50,
        validation_data=validation_generator,
        validation_steps=800)

The above assumes you are doing binary classification (set under "class_mode") but you can also do "categorical", "sparse", "input" etc. If you need something else, you can also create your own generator, or adapt the Keras one.