文本文件中的单词集

Question

我正在尝试创建一组单词，这严格来说仅意味着.txt文件中的字母。 该txt文件包含所有可能的字符，包括不可打印的文本。

没有资源库或馆藏库。 Python 3

例如，给定的.txt文件读取

*eBooks$ Readable By Both Humans and By Computers, Since 1971**

*These# eBooks@ Were Prepared By Thousands of Volunteers!

我将需要我的集合包含

  {'eBooks', 'Readable', 'By', 'Both', 'Humans', 'and', 'Computers', 'Since', 'These', 'Were', 'Prepared', 'Thousands', 'of', 'Volunteers'}

这是我所做的，但我的电视机上仍出现特殊字符和数字。 我只想要字母

import string
filecontent = []
word_set = {}
with open ("small.txt") as myFile:
    for line in myFile:
        line = line.rstrip()
        line = line.replace("\t","")
        for character in line:
            if character in string.digits or character in string.punctuation:
                line = line.replace(character, "")
            if line != "":
                filecontent.append(line)
lowerCase = [x.lower() for x in filecontent]
word_set = {word for line in lowerCase for word in line.split()}

Answer 1

您可以执行以下操作：

>>> from string import punctuation
>>> def solve(s):
        for line in s.splitlines():
            for word in line.split():
                word = word.strip(punctuation)
                if word.translate(None, punctuation).isalpha():
                    yield word
...                 
>>> s = '''*eBooks$ Readable By Both Humans and By Computers, Since 1971**

*These# eBooks@ Were Prepared By Thousands of Volunteers!'''
>>> set(solve(s))
set(['and', 'Both', 'Since', 'These', 'Readable', 'Computers', 'Humans', 'Prepared', 'of', 'Were', 'Volunteers', 'Thousands', 'By', 'eBooks'])

如果您使用的是Python 3，则需要将str.translate部分替换为：

table = dict.fromkeys(map(ord, punctuation)) #add this at the top of function
...
if word.translate(table).isalpha():
    ...

Answer 2

这是使用re regex模块的解决方案。 它还提供了字数统计，但是如果您不希望这样做，可以使用键，或者将其换成一组。

text = """*eBooks$ Readable By Both Humans and By Computers, Since 1971**

*These# eBooks@ Were Prepared By Thousands of Volunteers!"""

import re

from collections import Counter

words = Counter()

regex = re.compile(r"[a-zA-Z]+")

matches = regex.findall(text)
for match in matches:
  words[match.lower()] += 1

print words

或者，如果您将其存储在文件中；

with open("fileName") as textFile:
  text = "".join(textFile.readLines()) #Necesary to turn the file into one long string, rather than an array of lines.
  matches = regex.findall(text)
  for match in matches:
    words[match.lower()] += 1

这使

Counter({'by': 3, 'ebooks': 2, 'and': 1, 'both': 1, 'since': 1, 'these': 1, 'readable': 1, 'computers': 1, 'humans': 1, '1971': 1, 'prepared': 1, 'of': 1, 'were': 1, 'volunteers': 1, 'thousands': 1})

Answer 3

如果我是你，我就用过re.findall

import re
s = '''*eBooks$ Readable By Both Humans and By Computers, Since 1971**
*These# eBooks@ Were Prepared By Thousands of Volunteers!'''
set(re.findall('[a-zA-Z]+',s))

输出

set(['and', 'Both', 'Since', 'These', 'Readable', 'Computers', 'Humans', 'Prepared', 'of', 'Were', 'Volunteers', 'Thousands', 'By', 'eBooks'])