如何删除json数据python中的特殊字符

Question

I am reading a set of data from a json file. Content of the json file looks like:

"Address":"4820 ALCOA AVE�            ",
"City":"VERNON�   "

As you can see that it contains a special character and white space at the end. While reading this json data, it is coming like:

'address': '4820 ALCOA AVE�             '
'city': 'VERNON�   '

I can remove the whitespace easily but I am not sure how can I remove the ¿½ . I do not have direct access to json file so cannot edit it and even if I had access to json file, I would talk couple of hours to edit the file. Is there any way in python we can remove this special characters. Please help. Thanks

Answer 1

Looks like somewhere upstream wasn't handling character encoding properly and ended up with replacement characters... You may need to keep an eye out in case it mangled more important parts of the text (eg. accented characters, non-English letters, emoji).

For the immediate problem, you can load the JSON data with the utf-8 encoding, then strip the character '\�' .

   value = value.strip().strip('\ufffd')

If the replacement characters also appear in the middle (and you want to delete them), you might want to use replace() instead.

    value = value.replace('\ufffd', '').strip()

Answer 2

you can use regexp

    import re
    address = re.sub(r"[^\x20-\x7E]", "", "4820 ALCOA AVE� ")
    print(address)