设置检查词袋的Python

Question

I have a text processing script that needs to check if a word is in a bag of words. I have to do this many (10000s) of times. I would think that the most efficient way to do this would be to define the set of words that I am checking for wordBag = set(['these', 'are', 'my', 'words']) and then do if word in wordBag: . I looked at the documentation and this is average case O(1) and worst case O(n). Is this just due to chaining in hashsets? Is there a more pythonic way of doing this?

Answer 1

It really depends on the size of your set of words. For any non-huge amount (probably up to tens of thousands) your approach is absolutely fine and totally pythonic. Simplicity is a virtue!

If you do have a vast amount of words in your bag, a prefix-tree (or "trie") approach will likely be best, for that you can check PyPI for existing implementations ( example ).

EDIT: The O(n) worst-case is only in the case of huge numbers of hash collisions as you suspected, but that's not usually a problem in practice. In any case, I would go with the simplest approach first and only look at more advanced solutions if you actually have performance or memory issues.