无法在Windows中处理字符串

Question

I have written a python 2.7 code in linux and it worked fine.

It uses

os.listdir(os.getcwd())

to read folder names as variables and uses them later in some parts.

In linux I used simple conversion trick to manually convert the non asci characters into asci ones.

str(str(tfile)[0:-4]).replace('\xc4\xb0', 'I').replace("\xc4\x9e", 'G').replace("\xc3\x9c", 'U').replace("\xc3\x87", 'C').replace("\xc3\x96", 'O').replace("\xc5\x9e", 'S') ,str(line.split(";")[0]).replace(" ", "").rjust(13, "0"),a)) 

This approach failed in windows. I tried

udata = str(str(str(tfile)[0:-4])).decode("UTF-8")
asci = udata.encode("ascii","ignore")

Which also failed with following

DEM¦-RTEPE # at this string error occured

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python27\lib\lib-tk\Tkinter.py", line 1532, in __call__
    return self.func(*args)
  File "C:\Users\benhur.satir\workspace\Soykan\tkinter.py", line 178, in SparisDerle
    udata = str(str(str(tfile)[0:-4])).decode("utf=8")
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa6 in position 3: invalid start byte

How can I handle such characters in windows?

NOTE:Leaving them UTF causes xlswriter module to fail, so I need to convert them to asci. Missing characters are not desirable yet acceptable.

Answer 1

Windows does not like UTF8. You probably get the folder names in the default system encoding, generally win1252 (a variant of ISO-8859-1).

That's the reason why you could not find UTF8 characters in the file names. By the way the exception says you found a character of code 0xa6 , which in win1252 encoding would be | .

It does not say exactly what is the encoding on your windows system as it may depends on the localization, but it proves the data is not UTF8 encoded.

Answer 2

How about this?
You can use this for optional .replace()
In the module of string , there is a set of characters that can be used..

>>> import string
>>> string.digits+string.punctuation
'0123456789!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
>>>