如何在多个标记化单词列表中计算10个最常见的单词

Question

I have a data-set with a lot of lists of lists of tokenized words. for example:

['apple','banana','tomato']
['tomato','tree','pikachu']

I have around 40k lists like those, and I want to count the 10 most common words from all of the 40k lists together.

Anyone have any idea?

Answer 1

You could flatten the nested list with itertools.chain and take the most common words using Counter and its most_common method:

from itertools import chain
from collections import Counter

l = ['apple','banana','tomato'],['tomato','tree','pikachu']

Counter(chain(*l)).most_common(10)
# [('tomato', 2), ('apple', 1), ('banana', 1), ('tree', 1), ('pikachu', 1)]

Answer 2

a solution using dictionary

arrays = [['apple','banana','tomato'],['tomato','tree','pikachu']]
d = dict()
for array in arrays:
    for item in array:
        if item in d:
            d[item] += 1
        else:
            d[item] = 1
print(sorted( ((v,k) for k,v in d.items()), reverse=True)[:10])

Output

[('tomato', 2), ('apple', 1), ('banana', 1), ('tree', 1), ('pikachu', 1)]

Answer 3

I suggest to merge your lists to a one list eg

list_of_lists = [['apple','banana','tomato'],['tomato','tree','pikachu']]

import itertools
flat_list = list(itertools.chain(*list_of_lists))

Then use Counter to calculate your tokens and pick just the top 10

from collections import Counter
counter_of_flat_list = Counter(flat_list)

print(counter_of_flat_list.most_common(10)) # print top 10

[('tomato', 2), ('apple', 1), ('banana', 1), ('tree', 1), ('pikachu', 1)]