[英]Creating a dictionary for each word in a file and counting the frequency of words that follow it
I am trying to solve a difficult problem and am getting lost. 我正在努力解决一个棘手的问题并迷失方向。
Here's what I'm supposed to do: 这是我应该做的:
INPUT: file
OUTPUT: dictionary
Return a dictionary whose keys are all the words in the file (broken by
whitespace). The value for each word is a dictionary containing each word
that can follow the key and a count for the number of times it follows it.
You should lowercase everything.
Use strip and string.punctuation to strip the punctuation from the words.
Example:
>>> #example.txt is a file containing: "The cat chased the dog."
>>> with open('../data/example.txt') as f:
... word_counts(f)
{'the': {'dog': 1, 'cat': 1}, 'chased': {'the': 1}, 'cat': {'chased': 1}}
Here's all I have so far, in trying to at least pull out the correct words: 这是我到目前为止所做的一切,试图至少提出正确的话:
def word_counts(f):
i = 0
orgwordlist = f.split()
for word in orgwordlist:
if i<len(orgwordlist)-1:
print orgwordlist[i]
print orgwordlist[i+1]
with open('../data/example.txt') as f:
word_counts(f)
I'm thinking I need to somehow use the .count method and eventually zip some dictionaries together, but I'm not sure how to count the second words for each first word. 我想我需要以某种方式使用.count方法并最终将一些字典压缩在一起,但我不知道如何计算每个第一个单词的第二个单词。
I know I'm nowhere near solving the problem, but trying to take it one step at a time. 我知道我无法解决问题,但试图一步一步。 Any help is appreciated, even just tips pointing in the right direction.
任何帮助都表示赞赏,甚至只是指向正确方向的提示。
We can solve this in two passes : 我们可以在两个过程中解决这个问题
Counter
and count the tuples of two consecutive words using zip(..)
; Counter
并使用zip(..)
计算两个连续单词的元组; and Counter
in a dictionary of dictionaries. Counter
转为。 This results in the following code: 这导致以下代码:
from collections import Counter, defaultdict
def word_counts(f):
st = f.read().lower().split()
ctr = Counter(zip(st,st[1:]))
dc = defaultdict(dict)
for (k1,k2),v in ctr.items():
dc[k1][k2] = v
return dict(dc)
We can do this in one pass : 我们可以一次做到这一点:
defaultdict
as a counter. defaultdict
作为计数器。 So... For the sake of brevity, we'll leave the normalization and cleaning out: 所以......为了简洁起见,我们将保持标准化和清理:
>>> from collections import defaultdict
>>> counter = defaultdict(lambda: defaultdict(int))
>>> s = 'the dog chased the cat'
>>> tokens = s.split()
>>> from itertools import islice
>>> for a, b in zip(tokens, islice(tokens, 1, None)):
... counter[a][b] += 1
...
>>> counter
defaultdict(<function <lambda> at 0x102078950>, {'the': defaultdict(<class 'int'>, {'cat': 1, 'dog': 1}), 'dog': defaultdict(<class 'int'>, {'chased': 1}), 'chased': defaultdict(<class 'int'>, {'the': 1})})
And a more readable output: 更可读的输出:
>>> {k:dict(v) for k,v in counter.items()}
{'the': {'cat': 1, 'dog': 1}, 'dog': {'chased': 1}, 'chased': {'the': 1}}
>>>
Firstly that is some brave cat who chased a dog! 首先,那是一只追逐狗的勇敢的猫! Secondly it is a little tricky because we don't interact with this type of parsing every day.
其次,它有点棘手,因为我们不会每天与这种类型的解析交互。 Here's the code:
这是代码:
k = "The cat chased the dog."
sp = k.split()
res = {}
prev = ''
for w in sp:
word = w.lower().replace('.', '')
if prev in res:
if word.lower() in res[prev]:
res[prev][word] += 1
else:
res[prev][word] = 1
elif not prev == '':
res[prev] = {word: 1}
prev = word
print res
You could: 你可以:
zip(list_, list_[1:])
or any method that iterates by pairs; zip(list_, list_[1:])
或任何按zip(list_, list_[1:])
迭代的方法创建单词对; Like so: 像这样:
from collections import Counter
s="The cat chased the dog."
li=[w.lower().strip('.,') for w in s.split()] # list of the words
di={}
for a,b in zip(li,li[1:]): # words by pairs
di.setdefault(a,[]).append(b) # list of the words following first
di={k:dict(Counter(v)) for k,v in di.items()} # count the words
>>> di
{'the': {'dog': 1, 'cat': 1}, 'chased': {'the': 1}, 'cat': {'chased': 1}}
If you have a file, just read from the file into a string and proceed. 如果你有一个文件,只需从文件中读取一个字符串然后继续。
Alternatively, you could 或者,你可以
defaultdict
with a Counter
as a factory. Counter
的defaultdict
作为工厂。 Like so: 像这样:
from collections import Counter, defaultdict
li=[w.lower().strip('.,') for w in s.split()]
dd=defaultdict(Counter)
for a,b in zip(li, li[1:]):
dd[a][b]+=1
>>> dict(dd)
{'the': Counter({'dog': 1, 'cat': 1}), 'chased': Counter({'the': 1}), 'cat': Counter({'chased': 1})}
Or, 要么,
>>> {k:dict(v) for k,v in dd.items()}
{'the': {'dog': 1, 'cat': 1}, 'chased': {'the': 1}, 'cat': {'chased': 1}}
I think this is a one pass solution without importing defaultdict. 我认为这是一个单一的解决方案,没有导入defaultdict。 Also it has punctuation stripping.
它还有标点符号剥离。 I have tried to optimize it for large files or repeated opening of files
我试图为大文件或重复打开文件优化它
from itertools import islice
class defaultdictint(dict):
def __missing__(self,k):
r = self[k] = 0
return r
class defaultdictdict(dict):
def __missing__(self,k):
r = self[k] = defaultdictint()
return r
keep = set('1234567890abcdefghijklmnopqrstuvwxy ABCDEFGHIJKLMNOPQRSTUVWXYZ')
def count_words(file):
d = defaultdictdict()
with open(file,"r") as f:
for line in f:
line = ''.join(filter(keep.__contains__,line)).strip().lower().split()
for one,two in zip(line,islice(line,1,None)):
d[one][two] += 1
return d
print (count_words("example.txt"))
will output: 将输出:
{'chased': {'the': 1}, 'cat': {'chased': 1}, 'the': {'dog': 1, 'cat': 1}}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.