Python字典，其中包含每个键的值列表

Question

I have two different text files: One with words and their frequencies that looks like:

word1<space>frequency

Second one is a file that has a word in the first place followed by its associated features. It looks like:

word1<tab>feature1<tab>feature2................

Every word in the second file may have any number of features (ranging from 0-7 in my case)

For every word in file 1, I want all the features associated with it from file 2. I want to create a dictionary where the key is the word from file 1 and its corresponding value is a list of features obtained from file 2.

Also, I want unique features and want to eliminate duplicates from file 2 (I have not implemented it yet).

I have the following code but it gives the desired output only for the first word in file 1. mydict does contain all the other words from file 1 but they don't have any values associated with them.

mydict = dict()

with open('sample_word_freq_sorted.txt', 'r') as f1:
        data = f1.readlines()

with open('sample_features.txt', 'r') as f2:
        for item in data:
                root = item.split()[0]
                mylist = []
                for line in f2:
                        words = line.split()
                        if words[0] == root:
                                mylist.append(words[1:])
                mydict[root] = mylist

Also, the values for each key are different lists and not just one list which is not what I want. Can someone please help me with the bug is in my code?

Answer 1

mydict = dict()

with open('sample_word_freq_sorted.txt', 'r') as f1:
        data = set([ line.split()[0] for line in f1])

with open('sample_features.txt', 'r') as f2:
        for line in f2:
            word = line.split(' ')[0].strip()
            if word in data:
               mydict[word] = mydict.get(word,[]) + line.split(' ')[1:]

Answer 2

I think your most robust way would be to use Pandas and merge.

df1 = pd.read_csv('sample_word_freq_sorted.txt', delim_whitespace=True)
df2 = pd.read_csv('sample_features.txt', delimeter='\t')
df2 = df2.drop_duplicates()

df = df1.merge(df2, how='left', on='word')

Obviously that needs to be customized for the bits of your data not posted, but this would be much less prone to problems than trying to customize everything in a loop. It also handles your duplicate problem easily.

Whether this is the right solution also depends on what you want to do with the result - it may be that getting the dictionary version to work would be better in some situations.

Edit: When your data has no column headers, you can let Pandas just give them names, which will be integers starting with 0:

pd.read_csv(path, headers=None)

Then you can use the integers (eg df[0] will reference the first column named 0) or change the headers later, for example by assigning directly to df.columns = ['foo', 'bar', baz'] , or you can specify the headers in the loading:

pd.read_csv(path, names=['foo', 'bar', baz'])

Answer 3

A file is an iterator meaning you can only iterate over it once:

>>> x = (i for i in range(3)) #example iterator
>>> for line in x:
    print(line)

0
1
2
>>> for line in x: #second time produces no results.
    print(line)

>>> 

So the loop for line in f2: only produces values for the first time it is used (the first iteration of for item in data: ) To fix this you can either do f2 = f2.readlines() so you have a list that can be traversed more then once or find a way to construct your dictionary with only one iteration of f2 .

Then you get a list of sublists because you .append() each list of words to mylist , instead of .extend ing it by the additional words, so just changing:

mylist.append(words[1:])

to

mylist.extend(words[1:])

Should fix the other issue you are having.

---

This seems like a case where collections.defaultdict would come in handy, instead of going over the file many times adding items for each specific word the dict will automatically make empty lists for each new word, this would let you write your code something like this:

import collections
mydict = collections.defaultdict(list)

with open('sample_features.txt', 'r') as f2:
    for line in f2:
        tmp = line.split()
        root = tmp[0]
        words = tmp[1:]
        #in python 3+ we can use this notation instead of the above three lines:
        #root, *words = line.split()
        mydict[root].extend(words)

Although since you want to keep only unique features it would make more sense to use set s instead of list s since they -by definition- only contain unique elements, then instead of using .extend you would use .update :

import collections
mydict = collections.defaultdict(set)
   ....
        mydict[root].update(words)