字符串中隐藏的unicode字符集

Question

Some hidden set of Unicode characters appear in a string which needs to be removed.

I have a very large text which is extracted from a PDF file using PyPDF2 package. Now this extracted text has a lot of issues in it (like text in tables inside PDF which were structured will appear randomly when extracted) and lots of special characters also get embedded in it (like ~~~~~~~, }}}}}}}} etc) although those texts are not present when viewed as a PDF file. I tried removing those characters using the solution described in this , this and this link but the problem still appears

myText = "There is a set of hidden character here => <= but it will get printed in console"

print(myText)

Now I would like to have a clean text without those hidden characters.

Answer 1

The character \\x7f is the ascii character DEL , which explains why your attempts did not work. To remove all "special" ascii characters use this code:

See here for the bytes.decode documentation .

import string
a = b'There is a set of hidden character here =>\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f <= but i will get printed in console'
print(repr(a))
print(repr(''.join(i for i in a.decode('ascii', 'ignore') if i in string.printable)))

or this if no you don't want to import string:

a = b'There is a set of hidden character here =>\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f <= but i will get printed in console'
print(repr(a))
print(repr(''.join(i for i in a.decode('ascii', 'ignore') if 31 < ord(i) < 127 or i in '\r\n')))