Converting Hex NCR text representations to Unicode in Python

Question

I have a string parsed from a web page originally in chinese as:

若き葉末には風が立ち 森は翡翠の息を返す 雲の切れ間から注ぐ 光に君を見初めん

碧き瞳のほほえむとき そは鐘のひびき胸に打つ さしのべた腕に絡む 蔦の葉に君を逃す

残る　香り 水面をかけゆく恋の舟 つかの間の波に　揺られ

やさしき幻影　心に映るその姿よ 永遠なる君に　想いを捧げん

若き葉末には風は眠り 森は密やかに息を止む 抱きしめた腕のなかで 静かに君は消えゆく

月は　満ちて 黄金の羽根が舞いおちる 我はただひとり森に

祈りたまえや

However in the process of parsing it, it was converted into Hex NCR string in the following form:

若き葉末には風が立ち\n森は翡翠の息を返す\n雲の切れ間から注ぐ\n光に君を見初めん\n\n碧き瞳のほほえむとき\nそは鐘のひびき胸に打つ\nさしのべた腕に絡む\n蔦の葉に君を逃す\n\n残る　香り\n水面をかけゆく恋の舟\nつかの間の波に　揺られ\n\nやさしき幻影　心に映るその姿よ\n永遠なる君に　想いを捧げん\n\n若き葉末には風は眠り\n森は密やかに息を止む\n抱きしめた腕のなかで\n静かに君は消えゆく\n\n月は　満ちて\n黄金の羽根が舞いおちる\n我はただひとり森に\n\n祈りたまえや

I want to convert this string into its appropriate unicode format.

From my research I have been able to gather that for example 一 maps to the unicode string b'\\\一' .

This can be manually done by stripping &#x and prefixing a \\\\u\u003c/code> to the beginning of the string along with making the whole thing lowercase and converting to a bytestring by adding a b before the string. This is done in this repo but through the use of the inefficient eval function through code such as eval("b'\\\一") .

[EDIT: The above para is incorrect. It is not a bytestring but a unicode string as present in python2. The correct mapping would be 一 -> u'\一' ]

Is there a better way to do this? Considering edge cases where these hex map strings can be present in the middle of regular text such as here:

Je me levais tôt
Travailler en homme
Je me souviens du goût
Du café brûlant
Dans la tasse rouge
Et la femme qui dort
Les portes ouvertes de la grande usine
Bouffaient nos fils le jour de leurs quinze ans
On se levait tôt
Sortis de nos draps
On se retrouvait en bas
Les rues du village s'allumaient d'un coup
A six heures moins le quart
Les portes ouvertes de la grande usine
Bouffaient nos fils bien avant leurs quinze ans
On se lève trop tôt
On sait plus quoi faire
Dans le café des vieux
Les mains dans nos poches
Cachent nos poings noirs
Y'a plus qu'à qui change pas
Les portes sont fermées
Y'a plus de feu qui gronde
L'usine a tout vomi d'un seul coup
Pourquoi on fait ça
Pourquoi ça m'fait ça
Pourquoi on nous fait ça à nous

I am dealing with a large set of data where such characters can be strewn anywhere, and I need a meaningful way to deal with them.

So is there any better way to do this? Ideally one that is supported inherently by python.

If someone has a solution to my problem here, I will be immensely grateful. Thanks in advance.

Answer 1

Have a look at the html module in the standard library:

>>> import html
>>> html.unescape('Je me levais tôt')
'Je me levais tôt'
>>> html.unescape('若き葉末には')
'若き葉末には'

The result is a Unicode string (type str in Python 3). Note that the b'...' notation is for byte strings. The literal b'\\\一' in your example does not make much sense, since it is a byte string with 6 characters (\\, u, 4, e, 0, 0). You probably meant '\一' (or u'\一' in Python 2), which is a single-character Unicode string.