嘈杂文本的字符串匹配算法

Question

I have used OCR (optical character recognition) to get texts from images. The images contain book covers. Because of the images are so noisy, some characters are misrecognised, or some noises are recognised as a character.

Examples:

1. "w COMPUTER Nnwonxs i I "(Compuer Networks)
2. "s.ll NEURAL NETWORKS C "(Neural Networks)
3. "1llllll INFRODUCIION ro PROBABILITY ti iitiiili My "(Introduction of Probability)

I builded a dictionary with words, but i want to somehow match the recognised text with the dictionary. I tried LCS (Longest Common subsequence), but its not so effective.

What is the best string matching algorithm for this kind of problem? (So a part of string is just noise, but also the important part of string can has some misrecognised characters)

Answer 1

That's really a big question. Followings are something I know about it. For more details, you can read some related papers.

For single word, use Hamming Distance to calculate the similarity between the word your recognized by OCR and those in your dictionary;

this step is used to correct the the words have been recognized by OCR but do not exist.

Eg： If the result of OCR is INFRODUCIION which dosen't exist in your dictionary, you can find out the Hamming Distance of word 'INTRODUCTION' is 2. So it may be mis-recognized as 'INFRODUCIION'. However, the same word may be recognized as different words with the same Hamming Distance between them.

Eg： If the result of OCR is the CAY, you may find CAR and CAT are both with the same Hamming Distance of 1, so that will be confused.

In this case, there are several things can be used for analyze:

1. Still for single word, the image different between CAT and CAY is less that CAR and CAY. So for this reason, CAT seems the right word with a greater probability.

2. Then let us the context to caculate another probability. If the whold sentence is 'I drove my new CAY this morning', as for people usually drive a CAR but not a CAT, we have a better chance to regard the word CAY as CAR but not CAT.

3. For the frequency of the words used in the similar articles, use TF-TDF.

Answer 2

Are you saying you have a dictionary that defines all words that are acceptable?

If so, it should be fairly straight forward to take each word and find the closest match in your dictionary. Set a match threshold and discard the word if it does not reach the threshold.

I would experiment with the Soundex and Metaphone algorithms or the Levenshtein Distance algorithm .