Remove non Unicode characters from xml database with Python

Question

So I have a 9000 line xml database, saved as a txt, which I want to load in python, so I can do some formatting and remove unnecessary tags (I only need some of the tags, but there is a lot of unnecessary information) to make it readable. However, I am getting a UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 608814: character maps to <undefined> , which I assume means that the program ran into a non-Unicode character. I am quite positive that these characters are not important to the program (the data I am looking for is all plain text, with no special symbols), so how can I remove all of these from the txt file, when I can't read the file without getting the UnicodeDecodeError ?

Answer 1

One crude workaround is to decode the bytes from the file yourself and specify the error handling. EG:

for line in somefile:
    uline = line.decode('ascii', errors='ignore')

That will turn the line into a Unicode object in which any non-ascii bytes have been dropped. This is not a generally recommended approach - ideally you'd want to process XML with a proper parser, or at least know your file's encoding and open it appropriately (the exact details depend on your Python version). But if you're entirely certain you only care about ascii characters this is a simple fallback.

Answer 2

The error suggests that you're using open() function without specifying an explicit character encoding. locale.getpreferredencoding(False) is used in this case (eg, cp1252 ). The error says that it is not an appropriate encoding for the input.

An xml document may contain a declaration at the very begining that specifies the encoding used explicitly. Otherwise the encoding is defined by BOM or it is utf-8. If your copy-pasting and saving the file hasn't messed up the encoding and you don't see a line such as <?xml version="1.0" encoding="iso-8859-1" ?> then open the file using utf-8 :

with open('input-xml-like.txt', encoding='utf-8', errors='ignore') as file:
    ...

If the input is an actual XML then just pass it to an XML parser instead:

import xml.etree.ElementTree as etree

tree = etree.parse('input.xml')