深度学习 Denoisy Model 与 imageDataGenerator

Question

I have this code:

epochs =50
batch_size = 5
validation_split = 0.2


datagen = tf.keras.preprocessing.image.ImageDataGenerator(validation_split=validation_split )


train_generator = datagen.flow(
    X_train_noisy, y_train_denoisy, batch_size=batch_size, 
    subset='training'
)

val_generator = datagen.flow(
    X_train_noisy, y_train_denoisy, batch_size=batch_size,
    subset='validation'
)

history = model.fit(train_generator,
         steps_per_epoch=(len(X_train_noisy)*(1-validation_split)) // batch_size, epochs=epochs,
                    validation_data = val_generator, validation_steps=(len(X_train_noisy)*validation_split)//batch_size)

X_train_noisy and y_train_denoisy are ndarray ([20,512,512,1]) pe But I get this error:

training and validation subsets have different number of classes after the split

How can I solve that?

thanks!

Answer 1

probably what happened is when the data is split for training and validation, the set of files selected for validation did not include any files in one or more of the classes. This can happen when your data set is small. Try increasing the validation_split to a larger value say like.5 and see if the problem goes away. It should. Then reduce the size of the validation split until the error reoccurs. That will determine the minimum split value you can use. Remember the split is randomized so set the split value at something above the minimum value. Another (BETTER) alternative is to split the data using sklearn train_test_split. This function has a parameter stratify that splits the data but ensures that all classes are included in the two component. See code below

from sklearn.model_selection import train_test_split
X_train_noisy, X_valid_noisy, y_train_denoisy, y_valid_denoisy=train_test_split(X_train_noisy,
       y_train_denoisy, test_size=validation_split, 
      shuffle=True, random_state=123, 
      stratify=y_train_denoisy)

now use these split variable in model.fit