How can I recognize digits from photo (.jpg format) using Python and TF, Keras?

Question

I used OpenCV to crop images from the photo. From this:

to this :

Then I crop it to 5 different parts with different types of threshold and angle (in rotation matrix 2D) for training a neural network. Now I have 45 similar jpg files for any digits from 0 to 9. But I can't understand how could I train it with my own data, not using MNIST datasets Help me out to dealing with building a digit recognition program, please. I need to extract all the digits from img to text.

Answer 1

If you are going for the NN approach, I would first start with a small nn, and see how well it does, you can use the MNIST toy example from here .

Just note that you will need to use your own data, instead of mnist:

import tensorflow as tf

x_train, y_train = load_train_data()
x_test, y_test = load_test_data()


model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

Note that I 'invented' 2 functions: load_train_data() and load_test_data() , you need to implement them for your data, and return a tuple, of ((samples,x,y), labels) , for each one of the functions.

Once you get the feeling, I would explore a bit more advanced networks, you can look here: https://towardsdatascience.com/a-simple-2d-cnn-for-mnist-digit-recognition-a998dbc1e79a , its a nice tutorial for 2d CNN network, just use your data loading functions instead of the mnist.

As you will now probably face a wall, as you don't have enough data, you need to apply some data augmentation.
There is a very nice solution ' Deep Diffeomorphic Transformer Networks ' from the last CVPR, performs very well on digits classification with a low amount of samples. You can find the mnist code here , again use your functions for the data.