为什么我的神经网络预测在应用于 MNIST 手绘图像时是正确的，但在应用于我自己的手绘图像时却不正确？

Question

Background:

I am trying to create a basic neural network to recognize hand-drawn images using the MNIST dataset. I have things working when training/predicting against the MNIST data.

Goal:

I would like to start applying the model to non-MNIST images (ie hand-drawn images that I create myself).

Problem:

Every prediction of hand-drawn images that I create has ended up being incorrect (which is odd because predictions against MNIST images are 95% accurate).

Code:

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import cv2

mnist = tf.keras.datasets.mnist # 28x28 images of handwritten digits (0-9)

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = tf.keras.utils.normalize(x_train, axis=1)
x_test = tf.keras.utils.normalize(x_test, axis=1)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))

model.compile(optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy'])

model.fit(x_train, y_train, epochs=3)

val_loss, val_acc = model.evaluate(x_test, y_test)
print(val_loss, val_acc)

# prediction from MNIST dataset
index_of_mnist_img = 0
predictionsA = model.predict([x_test])
print(np.argmax(predictionsA[index_of_mnist_img]))
plt.imshow(x_test[index_of_mnist_img], cmap = plt.cm.binary)
plt.show()

# prediction from my own hand-drawn image (THIS IS WHERE THINGS START GOING WRONG)
img = cv2.imread('4.png')
img = cv2.resize(img, (28,28))
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img = np.reshape(img, [1,28,28])
predictionsB = model.predict(img)
print(np.argmax(predictionsB[0]))
plt.imshow(predictionsB[0])
plt.show()

Any ideas?

Answer 1

I believe that you need to invert the colormap for your new (hand-drawn) image.

When I look at MNIST example images, I see something like this:

# show mnist image
index_of_mnist_img = 0
plt.imshow(x_test[index_of_mnist_img], cmap = plt.cm.binary)
plt.show()

However, if I make an example hand-written digit, and read it in as you have, I see an inverted image.

img = cv2.imread("4.png")
img = cv2.resize(img, (28,28))
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
plt.imshow(img, cmap = plt.cm.binary)

You can invert the image with OpenCV by adding one line, cv2.bitwise_not() .

img = cv2.imread(r"4.png")
img = cv2.resize(img, (28,28))
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img= cv2.bitwise_not(img) # invert image
plt.imshow(img, cmap = plt.cm.binary)

When I invert the image, then I get correct predictions from the neural network you have trained above.

predictionsB = model.predict(img)
print(np.argmax(predictionsB[0]))

4

Answer 2

Do you need consider how to the train/test was created. This traning can be overrfited, with this, your give a good acurracity in train, but this is not trully whith test data.

Is also possible uses image augmentation to increase your dataset, because MNIST is a relative small dataset, and the numbers is centralized, the image does not have noise and etc.

And another concepts can be used, like Dropouts: The idea behind Dropouts is that they remove a random number of neurons in your neural network. This works very well for two reasons: The first is that neighboring neurons often end up with similar weights, which can lead to overfitting, so dropping some out at random can remove this.

Another thing is use Pooling Layers to reduce size of representation, with this it will detect more features.