CAPTCHAs Image Manipulation using Pillow

Question

As an exercise, I'm attempting to break the following CAPTCHA:

It doesn't seem like it would be too difficult to break as the edges seems to fairly solid and noise should be relatively easy to remove. Problem is, I have very little experience with image manipulation. Currently I'm using Python with the Pillow library to manipulate the CAPTCHA image, after which it will be passed into Tesseract for OCR.

In the following code I attempt to bring out the edges by sharpening the image and the convert the image to black and white

from PIL import Image, ImageFilter

try:
    img = Image.open("Captcha.jpg")
except:
    print("Can't load captcha.")
    exit()


# Bring out the edges by sharpening.    
out = img.filter(ImageFilter.SHARPEN)
out = out.convert("L")

out = out.point(lambda x: 0 if x<136 else 255, "1")

width, height = out.size
out = out.resize((width*5, height*5), Image.NEAREST)

out.save("captcha_modified.png")

At this point I see the following:

However, Tesseract is still unable to read the characters. As an experiment, I used good ol' mspaint to manually modify the image to a point to where it could be read by Tesseract:

So if can get the image to that point, I think Tesseract will do a fairly good job at detecting characters. So my current thoughts are that I need to enhance the edges and reduce the noise the image. Also, I imagine it would be easier for Tesseract to detect the letters if the letters will filled in rather than outlined, but I have not idea how I'd do this.

Any suggestions on how to go about this? Is there a better way to process the images?

Answer 1

I am short on time so this answer may not be incredibly useful but goes over my own 2 algorithms exactly. There isn't much code but a few method reccomendations. It is a good idea to use code rather than MS Paint.With code its actually really easy to break a captcha and achieve above 50% success rate. Behavioral recognition may be a better security mechanism or may be an additional one.

A. Edge Detection Method you use:

Edge detection really isn't necessary. In this case, just use the getpixel((x,y)) function and fill in the area between the bounding lines, recognizing to fill at lines 1,3,5;etc. and turn off the fill after intersection 2,4,6;etc. Luckilly, you chose an easy Captcha so edge detection is a decent solution without decluttering,rotating, and re-alignment.

B. Manipulation Method:

Another method I use utilizes OpenCV and pillow as well. I am really busy but am posting a blog article on this later at druid5.wordpress.com/ which will contain code examples of this method. Since it isn't illegal to get through them, at least I am told, I use the method I will post to collect data all the time. Mostly, contrast and detail from pillow, some basic clutter removal with stats, re-alignment with a basic dfs, and rotation (performable with opencv or easily with a kernal). Tesseract is a good choice for open source but it isn't too hard to create an OCR with opencv either.

This exercies is a decent introduction to OpenCV, PIL (pillow), image manipulation with math, and some other things that help with everything from robotics to AI.

Using flow control to find the failed conditions and try different routes may be necessary but the aim should always be a generic solution.