在python中创建一个词袋矢量的快速方法

Question

I have a corpus which has been sentence tokenized and word tokenized. Working in python, I took the 9,999 most common words and replaced out of vocabulary words with a special 'UNK', so that I have a vocabulary of 10,000 words and a python dictionary 'word_to_index' which maps each word to an integer.

I would like a binary bag-of-words representation, where the representation of each of the original sentences is a 10,000 dimension numpy vector of 0s and 1s. If a word i from the vocabulary is in the sentence, the index[ i ] in the numpy array will be a 1; otherwise, a 0. Until now, I've been using the following code:

def bag_of_words(sent, vocab_length, word_to_index):
    words = []
    rep = np.zeros(vocab_length)
    for w in sent:
        if w not in words:
            rep += np.eye(vocab_length)[word_to_index[w]]
            words.append(w)
    return rep

def get_bag_of_words_corpus(corpus, vocab_length, word_to_index):
    return np.array([bag_of_words(sent, vocab_length, word_to_index) for sent in corpus])

The problem is that for each sentence, it takes nearly 1 second to create the numpy vector. Seeing as my corpus is 12.2 M sentences I'd rather not wait ~4.7 months that it would take to process it. Can anyone give me any advice on speeding up this code. I thought about trying a smarter hashing technique, but I'm not sure that will give me the improvement I'm looking for.

Answer 1

Why are you creating a complete eye array?

Simply do

for w in sent:
    if w not in words:
        ind=word_to_index[w]
        rep[ind]+=1
        #rep += np.eye(vocab_length)[word_to_index[w]]
        words.append(w)

You can also try casting a sentence to sets.Set to eliminate duplicates. You should also use sets.Set for words, since the in function runs in O(1) if you are using Set.

Source