非ASCII字符消除而无需更改字符数

Question

I have data in fixed-width format I'd like to convert to CSV/tab delimited in python using only ASCII characters. I know very little about encodings, and some of the characters in the original file are non-ASCII. I can replace these with placeholders easily enough (I don't really care what they are), but this throws the character count off. I've tried subsequently replacing each sequence of more than 1 placeholder with a single placeholder, but there are some situations where the special characters occur in sequence.

I don't know what encoding was used for the original file, but I wouldn't be surprised if it was copy/pasted from MS word and features characters like ½, « etc.

For example, consider the following file test.txt which contains fields of length 1, 2 and 1, separated by a space (including trailing new line):

1 AA A
2 BB B
3 ¾  C
4 «¾ D
5 C  E

The simple python script:

with open('./test.txt', 'r') as f:
    for line in f:
        print len(line)

outputs

7
7
8
9
7

I've tried replacing the offending characters, but since they're read as two, this results in two placeholders inserted. I can then replace multiple placeholders with a single placeholder... but then consecutive placeholders throw the count.

import re
r = re.compile(r'\?\?+')

with open('./test.txt', 'r') as f, \
   open('./test_out1.txt', 'w') as w1, \
   open('./test_out2.txt', 'w') as w2:
  for line in f:
    q1 = line.decode('ascii', 'replace').replace(u'\ufffd', '?')
    w1.write(q1)
    q2 = r.sub('?', q1)
    w2.write(q2)

Results: test_out1.txt

1 AA A
2 BB B
3 ??  C
4 ???? D
5 C  E

test2.txt

1 AA A
2 BB B
3 ?  C
4 ? D
5 C  E

This obviously will also have issues if there's ever an actual '?' character next to a non-ASCII character in the source.

Am I missing a really simple way to do this?

Thanks in advance.

Answer 1

Given that your simple python script outputs different line lengths, you are dealing with a multi-byte encoding of some description.

The best approach would be to determine the encoding of the file. If the data is supposed to be fixed-width, this will be an encoding where every line is the same number of characters (as opposed to bytes).

For example:

$ cat test.txt
1 AA A
2 BB B
3 ¾  C
4 «¾ D
5 C  E

$ python3
Python 3.5.0
>>> with open("test.txt", "r", encoding="utf-8") as f:
...     for line in f:
...         print(len(line))
... 
7
7
7
7
7

If you get different lengths with utf-8 , try other multi-byte encodings until you find the right one. Once you've determined the input encoding, you can easily output the file with the non-ASCII characters replaced with placeholders:

$ python3
Python 3.5.0
>>> with open("test.txt", "r", encoding="utf-8") as infile:
...     with open("output.txt", "w", encoding="ascii", errors="replace") as outfile:
...         for line in infile:
...             outfile.write(line)

$ cat output.txt 
1 AA A
2 BB B
3 ?  C
4 ?? D
5 C  E