Loss always go to nan when transfer learning by ResNet50 in keras

Question

I am using transfer learning to train a image classifier by ResNet50 model in Keras and by loading the pre-trained weights, but the loss go to nan initially and instantly and the acc stay in random level.

Actually, I don't know what is wrong because I have used this model to train a classifier successfully, though it was not with high acc but it works well. This time it failed.

I tuned the lr but nothing happened. Someone said the data may have problems, so I changed the data and only to find with different images the same model will show different results(that is to say, some data/images works well and another data/images will results loss:nan instantly). How could that be? I am really confused and can't figure out what is wrong with my images.

Dataset: 8 classes and each class contains about 300 images.

Here is the code for all:

import keras
import h5py
import numpy as np
import matplotlib.pyplot as plt

from keras.applications import ResNet50
from keras.models import Sequential
from keras.layers import Dense, Flatten, GlobalAveragePooling2D
from keras.applications.resnet50 import preprocess_input
from keras.preprocessing.image import ImageDataGenerator


data_generator = ImageDataGenerator(preprocessing_function= preprocess_input, 
                        rescale = 1./255)

train_generator = data_generator.flow_from_directory("image/train", 
                        target_size = (100, 100), 
                        batch_size = 32, 
                        class_mode = "categorical")
dev_generator = data_generator.flow_from_directory("image/dev", 
                        target_size = (100, 100), 
                        batch_size = 32, 
                        class_mode = "categorical")

num_classes = 8
model = Sequential()
model.add(ResNet50(include_top = False, pooling = "avg", weights= "resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5"))
model.add(Dense(num_classes, activation = "softmax"))
model.layers[0].trainable = False

model.compile(optimizer= "adam", loss= "categorical_crossentropy", metrics=["accuracy"])

model.fit_generator(train_generator, steps_per_epoch= 1, epochs = 1)

and the running output is:

Epoch 1/1
1/1 [==============================] - 6s 6s/step - loss: nan - acc: 0.0938

Answer 1

First Correct “image/dev” to "image/dev"

I think your error lies on this line:

data_generator = ImageDataGenerator(preprocessing_function= preprocess_input, rescale = 1./255)

you double scale your data when you are using both the preprocess_input function and rescale = 1./255 . Try Removing rescaling ...

data_generator = ImageDataGenerator(preprocessing_function= preprocess_input)