用电脑理解手写数字

Question

i would like to ask you one question: wanted to implement a code which clarifies a picture done by hand ( by pen), let us consider such image

it is done by blue pen, which should be converted to the gray scale image using following code

from PIL import Image

user_test = filename
col = Image.open(user_test)
gray = col.convert('L')
bw = gray.point(lambda x: 0 if x<100 else 255, '1')
bw.save("bw_image.jpg")
bw
img_array = cv2.imread("bw_image.jpg", cv2.IMREAD_GRAYSCALE)
img_array = cv2.bitwise_not(img_array)
print(img_array.size)
plt.imshow(img_array, cmap = plt.cm.binary)
plt.show()
img_size = 28
new_array = cv2.resize(img_array, (img_size,img_size))
plt.imshow(new_array, cmap = plt.cm.binary)
plt.show()

idea is that i am taking image from camera directly, but it is losing structure of digit and comes only empty and black picture, like this

therefore computer can't understand which digit it is and neural networks fails to predict its label correctly, could you please tell me which transformation should i apply in order to detect this image much more precisely?

edit : 

i have apply following code

from PIL import Image

user_test = filename
col = Image.open(user_test)
gray = col.convert('L')
plt.hist(img_array)
plt.show()

and got

Answer 1

You have several issues here, and you can methodically address them. First of all you're having an issue with thresholding properly.

As I suggested in earlier comments, you can easily see why your original thresholding was unsuccessful.

import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from matplotlib import cm

im = Image.open('whatever_path_you_choose.jpg').convert("L")
im = np.asarray(im)
plt.hist(im.flatten(), bins=np.arange(255));

Looking at the image you gave:

Clearly the threshold should be somewhere between 100-200, not as in your original code. Also note that this distribution isn't very bimodal - so I'm not sure otsu's method would work well here.

If we eyeball it (this can be tuned), we can see that thresholding at 145-ish gives decent results in terms of segmentation.

im_thresh = (im >= 145)
plt.imshow(im_thresh, cmap=cm.gray)

Now you might have an additional issue that you have horizontal lines, you can address this by writing on blank paper as suggested. This wasn't exactly your question but I will try to address it anyways (in a naive fashion). You can try a naive solution of using a sobel filter (think of it as the derivative of the image to get the lines), followed by a median filter to get the approximately most common pixel intensity - the size of the filter might have to vary for different digits though. This should clear up some of the lines. For a more rigorous approach try reading up on hough line transform for detecting horizontal lines and try to whiten them out.

This is my very naive approach:

from skimage.filters import sobel
from scipy.ndimage import median_filter
#Sobel filter reverses intensities so subtracting the result from 1.0 turns it back to the original
plt.imshow(1.0 - median_filter(sobel(im_thresh), [10, 3]), cmap=cm.gray)

You can try cropping automatically afterwards. Honestly I think most neural networks that could recognize MNIST-like digits could recognize the result I posted at the end as well.

Answer 2

Try using skimage package like this. This has inbuilt functions for image processing:

from skimage import io

from skimage.restoration import denoise_tv_chambolle
from skimage.filters import threshold_otsu


image = io.imread('path/to/your/image', as_gray=True)

# Denoising
denoised_image = denoise_tv_chambolle(image, weight=0.1, multichannel=True)

# Thresholding
threshold = threshold_otsu(denoised_image)
thresholded_image = denoised_image > threshold