python IO：将单词从文本文件拆分为python数组，避免转义char，换行符，十六进制值

Question

I'm having a hard time importing and splitting words properly from a simple txt-file into a python array.

Txtfile:

test1 file test1 test1 
test2 test2 
test 3 test3, test3.

test 4, test 4.


test 5


^Ltest 6.

When doing a simple for lines in file: array.append(lines) this is the final array i recieve:

['test1 file test1 test1\xc2\xa0\n', 'test2 test2\xc2\xa0\n', 'test 3 test3, test3.\n', '\n', 'test 4, test 4.\n', '\n', '\n', 'test 5\n', '\n', '\n', '^Ltest 6.\xc2\xa0\n']

I'd like it to be something like this, where i have one item per actual english word or escaping character, and without the \\x__ hex substrings:

['test1', 'file', 'test1', 'test1', '\n', 'test2', 'test2', '\n', 'test', '3', 'test3', 'test3', '.', '\n', '\n', 'test', '4', 'test', '4', '.', '\n', '\n', '\n', 'test', '5', '\n', '\n', '\n', 'test 6', '.', \n']

Help would be really appreciated, thanks beforehand.

Answer 1

Single line solution:

[w for w in file.read().split() if re.match(r'[\w\n\.]+$',w)]

For parsing large file, it's better to pre-compile the regex.

import re
word_ptn = re.compile(r'[\w\n\.]+$')
[w for w in file.read().split() if word_ptn.match(w)]

The above regex will filter out strings like 'test1\\xc2\\xa0\\n' . If you want to keep it, extract the matched string from the regex results:

word_ptn = re.compile(r'[\w\n\.]+')
Lp = (word_ptn.match(w) for w in file.read().split())
[ w.group(0) for w in Lp if w ]