使用正则表达式在Python中拆分句子

Question

I'm trying to split words, punctuation, numbers from a sentence. However, my code produces output that isn't expected. How can I fix it?

This is my input text (in a text file):

 "I 2changed to ask then, said that mildes't of men2,

And my code outputs this:

['"', 'I', '2', 'changed', 'to', 'ask', 'then', ',', 'said', 'that', "mildes't", 'of', 'men2']

However, the expected output is:

 ['"', 'I', '2', 'changed', 'to', 'ask', 'then', ',', 'said', 'that', "mildes't", 'of', 'men','2']

Here's my code:

import re
newlist = []
f = open("Inputfile2.txt",'r')
out = f.readlines()
for line in out:
    word = line.strip('\n')
    f.close()
    lst = re.compile(r"\d|\w+[\w']+|\w|[^\w\s]").findall(word)
print(lst)

Answer 1

In regular expressions, '\\w' matches any alphanumeric character, ie [a-zA-Z0-9].

Also in the first part of your regular expression, it should be '\\d+' to match more than one digits.

The second and the third part of your regular expression '\\w+[\\w']+|\\w' can be merged into a single part by changing '+' to '*'.

import re
with open('Inputfile2.txt', 'r') as f:
    for line in f:
        word = line.strip('\n')
        lst = re.compile(r"\d+|[a-zA-Z]+[a-zA-Z']*|[^\w\s]").findall(word)
        print(lst)

This gives:

['"', 'I', '2', 'changed', 'to', 'ask', 'then', ',', 'said', 'that', "mildes't", 'of', 'men', '2', ',']

Note that your expected output is incorrect. It is missing a ','.