Python list comprehension “too many values to unpack”

Question

sorry for this question but I'm drive crazy with the error "too many values to unpack". This is the code

FREQ = 3
fourgrams=""
n = 4
tokens = token_text(text) # is a function that tokenize
fourgrams = ngrams(tokens, n)
final_list = [(item,v) for item,v in nltk.FreqDist(fourgrams) if v > FREQ]
print final_list

Where is the error? Thanks a lot

Answer 1

FreqDist is a dictionary-like object. Iterating it yields keys (not key-value pairs). If you want to iterate both key-value pairs, use FreqDist.items or FreqDist.iteritems :

final_list = [(item,v) for item,v in nltk.FreqDist(fourgrams).items() if v > FREQ]

Answer 2

Take a look at this:

from collections import Counter

from nltk.corpus import brown
from nltk.util import ngrams

# Let's take the first 10000 words from the brown corpus
text = brown.words()[:10000]
# Extract the ngrams
bigrams = ngrams(text, 2)
# Alternatively, unstead of a FreqDist, you can simply use collections.Counter
freqdist = Counter(bigrams)
print len(freqdist)
# Gets the top 5 ngrams
top5 = freqdist.most_common()[:5]
print top5
# Limits v > 10
freqdist = {k:v for k,v in freqdist.iteritems() if v > 10}
print len(freqdist)

[out]:

7615
[(('of', 'the'), 95), (('.', 'The'), 76), (('in', 'the'), 59), (("''", '.'), 40), ((',', 'the'), 36)]
34