根据图像中的坐标对 OCR-Result 的单独字符串进行分组

Question

I use easyocr to read the key figures from an image (display output of measuring instrument). Because of different proportions of characters on the picture, some characters/strings, that are meant to be one unit, like value and unit (eg "230 Volt"), are recognised as separate strings ("230", "Volt"). Another example are multiline strings, where each line is recognised as separate string. To illustrate it I prepared a picture. Its a little bit exaggerated but I hope it´s easy to understand.

Example picture to illustrate the problem

What I want to do

I try to find the elements that are on the same line or column (and very close to each other) and concatenate these strings.

Example data to handle (Output of easyocr)

(Coordinates are from top-left corner to bottom-left corner in clockwise direction)

([[239, 31], [563, 31], [563, 195], [239, 195]], '230', 0.7262734770774841)
([[591, 147], [661, 147], [661, 183], [591, 183]], 'Volt', 0.983400155647826)
([[801, 171], [1039, 171], [1039, 239], [801, 239]], 'This is a', 0.9870205241250117)
([[802, 256], [1232, 256], [1232, 328], [802, 328]], 'sentence with', 0.9997852752308181)
([[805, 341], [1065, 341], [1065, 427], [805, 427]], 'multiple', 0.9999849956753041)
([[212, 427], [311, 427], [311, 479], [212, 479]], 'Text', 0.9999873638153076)
([[362, 428], [474, 428], [474, 476], [362, 476]], 'More', 0.9999922513961792)
([[505, 413], [643, 413], [643, 479], [505, 479]], 'Text', 0.9999755620956421)
([[798, 428], [1136, 428], [1136, 500], [798, 500]], 'linebreaks.', 0.8525006562415545)
([[317, 601], [479, 601], [479, 669], [317, 669]], 'More', 0.9999911785125732)
([[529, 603], [665, 603], [665, 669], [529, 669]], 'Text', 0.9757571413464591)
([[699, 603], [841, 603], [841, 669], [699, 669]], 'with', 0.9999924302101135)
([[950, 608], [1182, 608], [1182, 683], [950, 683]], 'spaces.', 0.8026406194725301)

Output as DataFrame

I tried to handle it as Dataframe and split the values to x and y for each point. I though this view will help me. But i am still stucked

             Text     Score  tl_x  tl_y  tr_x  tr_y  bl_x  bl_y  br_x  br_y
0             230  0.726273   239    31   563    31   239   195   563   195
1            Volt  0.983400   591   147   661   147   591   183   661   183
2       This is a  0.987021   801   171  1039   171   801   239  1039   239
3   sentence with  0.999785   802   256  1232   256   802   328  1232   328
4        multiple  0.999985   805   341  1065   341   805   427  1065   427
5            Text  0.999987   212   427   311   427   212   479   311   479
6            More  0.999992   362   428   474   428   362   476   474   476
7            Text  0.999976   505   413   643   413   505   479   643   479
8     linebreaks.  0.852501   798   428  1136   428   798   500  1136   500
9            More  0.999991   317   601   479   601   317   669   479   669
10           Text  0.975757   529   603   665   603   529   669   665   669
11           with  0.999992   699   603   841   603   699   669   841   669
12        spaces.  0.802641   950   608  1182   608   950   683  1182   683

What output I want to achieve:

I`m happy with a list of concatinated strings

["230 Volt", "This is a sentence with multiple linebreaks.","More text with spaces", ...]

What I tried

• I tried to sort the column-values with exact or almost the same values to bins with pd.cut(). And access the strings over the bin-name, but I have no idea how to code it, so its not hardcoded and can be run automatically on different pictures.
• Also tried to use np.isclose() with relative tolerance to values to group them together
• tried looping with a lot of if-conditions, but nothing works.

Help

I am sure there is a very simple solution to this, just I am not good enough at programming yet to see it.

What would be the best approach to find the closest neighbours (on the same line/column) and group them together?

Answer 1

Every time you will similar pattern text like in above case you get 230 volts ...Like in another example you will get 320 volts? ...So which will be formatted as x volts?

If so

import pandas as pd
c1_str = ' '.join(df["Text"])

c1_str =  c1_str.replace('linebreaks', 'linebreaks|')
c1_str =  c1_str.replace('Volt', 'Volt|')
mask =c1_str.split('|')
print(mask)

Gives #

['230 Volt', ' This is a sentence with multiple linebreaks', '. More text with spaces']

Explination:

Convert df column to string & manuplate string according to pattern you want by creating split patterns. Converting string to list based on split pattern |