how to make accuracy/loss plot in one plot if CNN model trained both on cifar10/100 in keras?

Question

using benchmark image dataset like cifar10 , cifar100 in convolutional NN is standard practice, and I am wondering is there any way we can both use cifar10 , cifar100 on same configuration of convolutional NN, so we could see training accuracy/loss of CNN model on two different dataset. Because it is time consuming run exactly same model on two different benchmark dataset separately. I am not sure how to achieve this in keras.

I think we might run the code in parallel and encapsulate the training model output in one dictionary then make accuracy/loss plot in one plot. I tried few different way of achieving this, I always get stuck with parallel run of CNN on cifar10 , and cifar100 . I used nivdia GPU to run experiment, but not sure to run parallel of CNN on feeding two different dataset? Is there any way to make this happen easily in tensorflow/keras? Is that doable? Any possible thoughts?

my current attempt

here is my current attempt where I haven't completely figured out how to run CNN on cifar10 and cifar100 in parallel and collect all training output in dictionary for making accuracy/loss plot.

import tensorflow as tf
import keras
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0

nb_classes = 10
y_train = to_categorical(y_train, nb_classes)
y_test = to_categorical(y_test, nb_classes)

def cnn(drop_rate=0.1):
    model = Sequential()
    model.add(Conv2D(32, (3, 3), input_shape=(32, 32, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Flatten())
    model.add(Dense(64))
    model.add(Activation('relu'))
    model.add(Dropout(drop_rate))
    model.add(Dense(10))
    model.add(Activation('sigmoid'))
    model.compile(loss='binary_crossentropy',optimizer='rmsprop',metrics=['accuracy'])
    return model

conv_model = cnn()
history = conv_model.fit(X_train, y_train, shuffle=True, callbacks=[callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=0, mode='min')],epochs=10, verbose=1, batch_size=128, validation_data=(X_test, y_test),validation_split=0.2)

## example plotting
plt.figure(figsize=(20,10))
plt.plot(history.history['loss'], 'r', linewidth=3)
plt.plot(history.history['val_loss'], 'b', linewidth=3)
plt.legend(['training loss', 'validation_loss'], fontsize=12)
plt.xlabel('epochs', fontsize=12)
plt.ylabel('loss', fontsize=12)

if we could able to run CNN model by feeding cifar10 , cifar100 in parallel, and expecting all training output will be in history , then we could access history.history to make plot of corresponding accuracy/loss of cifar10 and cifar100 in two subplot (one for accuracy, one for loss).

How to achieve this easily? Is there any efficient way to achieve this in tensorflow? Any possible thoughts to make this happen? Thanks

update

I was thinking we could do this in for loop but not sure it is good idea to do this. I bet there must be efficient and elegant way of doing this. Can anyone point me out how to achieve this as I expected? Any idea?

Answer 1

You cannot run the models in parallel, unless you would:

1. Have one trained on CPU and the other on GPU. This will still represent a problem since some pre-processing operations are still done on CPU and you may very well run into a bottleneck.
2. You have two GPUs and you train one on the first GPU and the second model on the second GPU.

The solution to your problem is to train sequentially, serialize the history objects(use pickle), and then load both history objects and plot on the same plot.

PS: You must use softmax and categorical_crossentropy for multi-class classification, instead of sigmoid and binary_crossentropy .

(1) For using pickle:

with open('filename.pickle', 'wb') as handle:
    pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL)

with open('filename.pickle', 'rb') as handle:
    b = pickle.load(handle)

(2) For enabling specific GPUs to be "seen" by the python process.

Eg: In train_cifar_100.py

import os
os.environ["CUDA_VISIBLE_DEVICES"]="0,1"
#train
#serialize

Eg In train_cifar_10.py

import os
os.environ["CUDA_VISIBLE_DEVICES"]="2,3"
#train
#serialize