Why there's a bad accuracy on dataset when it's used both for validation and training?

Question

I trained a model with ResNet50 and got an amazing accuracy of 95% on training set. I took the same training set for validation and the accuracy seem very bad.(<0.05%)

from keras.preprocessing.image import ImageDataGenerator

train_set = ImageDataGenerator(horizontal_flip=True,rescale=1./255,shear_range=0.2,zoom_range=0.2).flow_from_directory(data,target_size=(256,256),classes=['airplane','airport','baseball_diamond',
                                                                    'basketball_court','beach','bridge',
                                                                  'chaparral','church','circular_farmland',
                                                                  'commercial_area','dense_residential','desert',
                                                                  'forest','freeway','golf_course','ground_track_field',
                                                                  'harbor','industrial_area','intersection','island',
                                                                  'lake','meadow','medium_residential','mobile_home_park',
                                                                  'mountain','overpass','parking_lot','railway','rectangular_farmland',
                                                                  'roundabout','runway'],batch_size=31)

---

from keras.applications import ResNet50 
from keras.applications.resnet50 import preprocess_input
from keras import layers,Model

conv_base = ResNet50(
    include_top=False,
    weights='imagenet')


for layer in conv_base.layers:
    layer.trainable = False

x = conv_base.output
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x) 
predictions = layers.Dense(31, activation='softmax')(x)
model = Model(conv_base.input, predictions)

---

# here you will write the path for train data or if you create your val data then you can test using that too. 
# test_dir = ""
test_datagen = ImageDataGenerator(rescale=1. / 255) 
test_generator = test_datagen.flow_from_directory( 
    data, 
    target_size=(256, 256), classes=['airplane','airport','baseball_diamond',
                                                                    'basketball_court','beach','bridge',
                                                                  'chaparral','church','circular_farmland',
                                                                  'commercial_area','dense_residential','desert',
                                                                  'forest','freeway','golf_course','ground_track_field',
                                                                  'harbor','industrial_area','intersection','island',
                                                                  'lake','meadow','medium_residential','mobile_home_park',
                                                                  'mountain','overpass','parking_lot','railway','rectangular_farmland',
                                                                  'roundabout','runway'],batch_size=1,shuffle=True)

---

model.compile(loss='categorical_crossentropy',optimizer='Adam',metrics=['accuracy'])
model.fit_generator(train_set,steps_per_epoch=1488//31,epochs=10,verbose=True,validation_data = test_generator, 

validation_steps = test_generator.samples // 31)

Epoch 1/10
48/48 [==============================] - 27s 553ms/step - loss: 1.9631 - acc: 0.4825 - val_loss: 4.3134 - val_acc: 0.0208
Epoch 2/10
48/48 [==============================] - 22s 456ms/step - loss: 0.6395 - acc: 0.8212 - val_loss: 4.7584 - val_acc: 0.0833
Epoch 3/10
48/48 [==============================] - 23s 482ms/step - loss: 0.4325 - acc: 0.8810 - val_loss: 5.3852 - val_acc: 0.0625
Epoch 4/10
48/48 [==============================] - 23s 476ms/step - loss: 0.2925 - acc: 0.9153 - val_loss: 6.0963 - val_acc: 0.0208
Epoch 5/10
48/48 [==============================] - 23s 477ms/step - loss: 0.2275 - acc: 0.9341 - val_loss: 5.6571 - val_acc: 0.0625
Epoch 6/10
48/48 [==============================] - 23s 478ms/step - loss: 0.1855 - acc: 0.9489 - val_loss: 6.2440 - val_acc: 0.0208
Epoch 7/10
48/48 [==============================] - 23s 483ms/step - loss: 0.1704 - acc: 0.9543 - val_loss: 7.4446 - val_acc: 0.0208
Epoch 8/10
48/48 [==============================] - 23s 487ms/step - loss: 0.1828 - acc: 0.9476 - val_loss: 7.5198 - val_acc: 0.0417

What could be the reason?!

Answer 1

You have assigned train_set and test_datagen differently. In particular one is flipped and scaled where the other isn't. As I mentioned in my comment, if its the same data it will have the same accuracy. You can see a model is overfitting when you use validation correctly and use unseen data for validation. Using the same data will always give the same accuracy for training and validation

Answer 2

not sure what is exactly wrong but it is NOT an over fitting issue. It is clear your validation data(same as training data) is not going in correctly. For one thing you set the validation batch size =1 but you set the validation steps as validation_steps = test_generator.samples // 31) . If test_generator,samples = 1488 then you have 48 steps but with a batch size of 1 you will only validate 48 samples. You want to set the batch size and steps so that batch_size X validation_steps equals the total number of samples. That way you go through the validation set exactly one time. I also recommend that for the test generator you set shuffle=False. Also why do you bother entering all the class names. If you have your class directories labeled as 'airplane','airport','baseball_diamond' etc then you don;t need to specifically define the classes flow from directory will do that for you automatically. See documentation below.

classes: Optional list of class subdirectories (eg ['dogs', 'cats']). Default: None. If not provided, the list of classes will be automatically inferred from the subdirectory names/structure under directory, where each subdirectory will be treated as a different class (and the order of the classes, which will map to the label indices, will be alphanumeric). The dictionary containing the mapping from class names to class indices can be obtained via the attribute class_indices.

Your training data is actually different than your test data because you are using data augmentation in the generator. That's OK it may lead to a small difference between your test and validation accuracy but your validation accuracy should be pretty close once you get the validation data to go in correctly