迭代列表中的元素以检查是否在字典 python 和 numpy 中

Question

I want to use numpy to speed up my computation where in I have a dictionary and I want to create a vector from it based on the presence of words as keys in the dictionary. I currently do this - a dummy example is provided for better understanding, the actual data is much larger:

self.bigram_freq =  {"a cat":3, "man child"2, "pokemon team":4} 
sentences = ['a boy ran over a cat with his bike yesterday afternoon']
for sentence in sentences:
    feature_vector = []
    # generate pairs from this sentence and see if in bigram_freq
    bigram_pairs = self.retrieve_pairs(sentence)
    dummy_dict = dict.fromkeys(self.bigram_freq, 0)
    for pair in bigram_pairs:
        if pair in self.bigram_freq:
            dummy_dict[pair] +=1
    feature_vector = list(dict(sorted(dummy_dict.items(), key=lambda item: item[0])).values())
            outputVector.append(feature_vector)

But due to the two loops, its a lot slower. I was wondering if this could be sped up using numpy and np.where. I was thinking of creating an array of np.zeros and then populating a specific index of the ndarray when the corresponding token (a pair from bigram_pairs) is present but I am unable to do so. Any help would be appreciated.

Answer 1

I have attempted to fix your script like this in order to make it work:

outputVector = []
bigram_freq =  {"a cat":3, "man child":2, "pokemon team":4} 
sentences = ['a boy ran over a cat with his bike yesterday afternoon', 
             'he was dreaming about pokemon team at the moment he hit a cat']
S = sentence.split(' ')
bigram_pairs = [f'{x} {y}' for x,y in zip(S[:-1], S[1:])] 
>>> bigram_pairs
['he was', 'was dreaming', 'dreaming about', 'about pokemon', 'pokemon team', 'team at', 'at the', 'the moment', 'moment he', 'he hit', 'hit a', 'a cat']

Now, in Python you do it like this:

for sentence in sentences:
    dummy_dict = dict.fromkeys(bigram_freq, 0)
    for pair in bigram_pairs:
        if pair in bigram_freq:
            dummy_dict[pair] +=1
    feature_vector = list(dict(sorted(dummy_dict.items(), key=lambda item: item[0])).values())
    outputVector.append(feature_vector)        
>>> outputVector
[[1, 0, 1], [1, 0, 1]]

And you want to make it faster. Now take a look at this question . You don't actually need to create a list of all pairs because numpy allows you to check if a specific pair is in a sentence:

outputVector = []
match_with = list(bigram_freq)
for sentence in sentences:
    feature_vector = np.core.defchararray.find(sentence, match_with)!=-1
    outputVector.append(feature_vector)
>>> outputVector
[array([ True, False, False]), array([ True, False,  True])]