“WARNING:tensorflow:Your input run out of data”训练时出现错误 Keras Model

Question

The full warning is: WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 3400 batches). You may need to use the repeat() function when building your dataset.

# importing libraries
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras import backend as K
import tensorflow as tf

train_data_dir = 'marvel/train'
validation_data_dir = 'marvel/valid'
nb_train_samples = 2584
nb_validation_samples = 451
epochs = 100
batch_size_train = 76
batch_size_val = 41

if K.image_data_format() == 'channels_first':
    input_shape = (3, 200, 200)
else:
    input_shape = (200, 200, 3)

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(200, 200, 3)),
    tf.keras.layers.MaxPooling2D(2, 2),
    tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(8, activation='softmax')
])

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                    target_size=(200, 200),
                                                    batch_size=batch_size_train,
                                                    classes=['black widow', 'captain america', 'doctor strange', 'hulk', 'iron man', 'loki', 'spiderman', 'thanos'],
                                                    class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(200, 200),
    batch_size=batch_size_val, class_mode='categorical')

model.fit(train_generator,
          steps_per_epoch=nb_train_samples // batch_size_train,
          epochs=epochs, validation_data=validation_generator,
          validation_steps=nb_validation_samples // batch_size_val)

model.save_weights('characterImg.h5')
print("Saved model characterImg.h5")

The above is my code. Can anyone help me understand what the error actually means? I'm having a lot of trouble with it. Thank you! (Let me know if you need more info)

Answer 1

Okay I'm not sure if this will work for everyone but to fix this I simply deleted the line

steps_per_epoch=nb_train_samples // batch_size_train,

and it worked. Everything I realise it's not ideal but for those looking for a desperate fix this might do you

Answer 2

Looks like your dataset length is less than your nb_train_samples / nb_validation_samples .

Add repeat() call before fitting:

train_generator = train_generator.repeat()
validation_generator = validation_generator.repeat()