如何在文本文件中搜索一组单词？

Question

I'm writing a project on extracting a semantic orientation from a review stored in a text file. I have a 400*2 array, each row contains a word and it's weight. I want to check which of these words is in the text file, and calculate the weight of the whole content.

My question is -

what is the most efficient way to do it? Should I search for each word separately, for example with a for loop? Do I get any benefit from storing the content of the text file in a string object?

Answer 1

https://docs.python.org/3.6/library/mmap.html

This may work for you. You can use find

Answer 2

This may be out of the box thinking, but if you don't care for semantic/grammatic connection of the words:

• sort all words from the text by length
• sort your array by length

.

• Write a for-loop:
• Call len() (length) on each word from the text.
• Then only check against those words which have the same length.

With some tinkering it might give you a good performance boost instead of the "naive" search.

Also look into search algorithms if you want to achieve an additional boost (concerning finding the first word (of the 400) with eg 6 letters - then go "down" the list until the first word with 5 letters comes up, then stop.

Alternatively you could also build an index array with the indexes of the first and last of all 5-letter words (analog for the rest), assuming your words dont change.