Iterating through multiple values in a dictionary?

Question

I have a list and dictionary of words:

word_list = ["it's","they're","there's","he's"]

And a dictionary containing information as to how frequently the words in words_list appear in several documents:

dict = [('document1',{"it's": 0,"they're": 2,"there's": 5,"he's": 1}),
('document2',{"it's": 4,"they're": 2,"there's": 3,"he's": 0}),
('document3',{"it's": 7,"they're": 0,"there's": 4,"he's": 1})]

I want to develop a data structure (data frame, perhaps?) that looks like the following:

file       word       count
document1  it's        0
document1  they're     2
document1  there's     5
document1  he's        1
document2  it's        4
document2  they're     2
document2  there's     3
document2  he's        0
document3  it's        7
document3  they're     0
document3  there's     4
document3  he's        1

I'm trying to find the words used most often in these documents. I have more than 900 documents.

I'm thinking of something like the following:

res = {}
for i in words_list:
    count = 0
    for j in dict.items():
         if i == j:
              count = count + 1
              res[i,j] = count

Where can I go from here?

Answer 1

Ok first things first, your dict is not a dict and should be built as one like so

d = {'document1':{"it's": 0,"they're": 2,"there's": 5,"he's": 1},
    'document2':{"it's": 4,"they're": 2,"there's": 3,"he's": 0},
    'document3':{"it's": 7,"they're": 0,"there's": 4,"he's": 1}}

now that we actually have a dictionary we can use pandas to build a dataframe but in order to get it the way you want we will have to build a list of lists out of the dictionary. Then we will create a dataframe and label the columns and then sort

import collections
import pandas as pd

d = {'document1':{"it's": 0,"they're": 2,"there's": 5,"he's": 1},
    'document2':{"it's": 4,"they're": 2,"there's": 3,"he's": 0},
    'document3':{"it's": 7,"they're": 0,"there's": 4,"he's": 1}}

d = pd.DataFrame([[k,k1,v1] for k,v in d.items() for k1,v1 in v.items()], columns = ['File','Words','Count'])
print d.sort(['File','Count'], ascending=[1,1])

         File    Words  Count
1   document1     it's      0
0   document1     he's      1
3   document1  they're      2
2   document1  there's      5
4   document2     he's      0
7   document2  they're      2
6   document2  there's      3
5   document2     it's      4
11  document3  they're      0
8   document3     he's      1
10  document3  there's      4
9   document3     it's      7

If you want the top n occurrences then you can use groupby() and then either head() or tail() when sorting

d = d.sort(['File','Count'], ascending=[1,1]).groupby('File').head(2)

         File    Words  Count
1   document1     it's      0
0   document1     he's      1
4   document2     he's      0
7   document2  they're      2
11  document3  they're      0
8   document3     he's      1

the list comprehension returns a list of lists that looks like this

d = [['document1', "he's", 1], ['document1', "it's", 0], ['document1', "there's", 5], ['document1', "they're", 2], ['document2', "he's", 0], ['document2', "it's", 4], ['document2', "there's", 3], ['document2', "they're", 2], ['document3', "he's", 1], ['document3', "it's", 7], ['document3', "there's", 4], ['document3', "they're", 0]]

in order to build the dictionary properly you would just use something along the lines of

d['document1']['it\'s'] = 1

If for some reason you are dead set on using the list of tuples of str's and dicts you can use this list comprehension instead

[[i[0],k1,v1] for i in d for k1,v1 in i[1].items()]

Answer 2

How about something like this?

word_list = ["it's","they're","there's","he's"]

frequencies = [('document1',{"it's": 0,"they're": 2,"there's": 5,"he's": 1}),
('document2',{"it's": 4,"they're": 2,"there's": 3,"he's": 0}),
('document3',{"it's": 7,"they're": 0,"there's": 4,"he's": 1})]

result = []
for document in frequencies:
    for word in word_list:
        result.append({"file":document[0], "word":word,"count":document[1][word]})

print result