在python列表中查找出现的快速方法

Question

I have a set of unique words called h_unique . I also have a 2D list of documents called h_tokenized_doc which has a structure like:

[ ['hello', 'world', 'i', 'am'], 
  ['hello', 'stackoverflow', 'i', 'am'], 
  ['hello', 'world', 'i', 'am', 'mr'], 
  ['hello', 'stackoverflow', 'i', 'am', 'pycahrm'] ]

and h_unique as:

('hello', 'world', 'i', 'am', 'stackoverflow', 'mr', 'pycharm')

what I want is to find the occurrences of the unique words in the tokenized documents list.
So far I came up with this code but this seems to be VERY slow. Is there any efficient way to do this?

term_id = []
for term in h_unique:
    print term
    for doc_id, doc in enumerate(h_tokenized_doc):
        term_id.append([doc_id for t in doc if t == term])

In my case I have a document list of 7000 documents, structured like:

[ [doc1], [doc2], [doc3], ..... ]

Answer 1

It'll be slow because you're running through your entire document list once for every unique word. Why not try storing the unique words in a dictionary and appending to it for each word found?

unique_dict = {term: [] for term in h_unique}
for doc_id, doc in enumerate(h_tokenized_doc):
    for term_id, term in enumerate(doc):
        try:
            # Not sure what structure you want to keep it in here...
            # This stores a tuple of the doc, and position in that doc
            unique_dict[term].append((doc_id, term_id))
        except KeyError:
            # If the term isn't in h_unique, don't do anything
            pass

This runs through all the document's only once.

From your above example, unique_dict would be:

{'pycharm': [], 'i': [(0, 2), (1, 2), (2, 2), (3, 2)], 'stackoverflow': [(1, 1), (3, 1)], 'am': [(0, 3), (1, 3), (2, 3), (3, 3)], 'mr': [(2, 4)], 'world': [(0, 1), (2, 1)], 'hello': [(0, 0), (1, 0), (2, 0), (3, 0)]}

(Of course assuming the typo 'pycahrm' in your example was deliberate)

Answer 2

term_id.append([doc_id for t in doc if t == term])

This will not append one doc_id for each matching term; it will append an entire list of potentially many identical values of doc_id . Surely you did not mean to do this.

Based on your sample code, term_id ends up as this:

[[0], [1], [2], [3], [0], [], [2], [], [0], [1], [2], [3], [0], [1], [2], [3], [], [1], [], [3], [], [], [2], [], [], [], [], []]

Is this really what you intended?

Answer 3

If I understood correctly, and based on your comment to the question where you say

yes because a single term may appear in multiple docs like in the above case for hello the result is [0,1, 2, 3] and for world it is [0, 2]

it looks like what you wanna do is: For each of the words in the h_unique list (which, as mentioned, should be a set , or keys in a dict , which both have a search access of O(1) ), go through all the lists contained in the h_tokenized_doc variable and find the indexes in which of those lists the word appears.

IF that's actually what you want to do, you could do something like the following:

#!/usr/bin/env python

h_tokenized_doc = [['hello', 'world', 'i', 'am'],
                   ['hello', 'stackoverflow', 'i', 'am'],
                   ['hello', 'world', 'i', 'am', 'mr'],
                   ['hello', 'stackoverflow', 'i', 'am', 'pycahrm']]

h_unique = ['hello', 'world', 'i', 'am', 'stackoverflow', 'mr', 'pycharm']

# Initialize a dict with empty lists as the value and the items 
# in h_unique the keys
results = {k: [] for k in h_unique}

for i, line in enumerate(h_tokenized_doc):
    for k in results:
        if k in line:
            results[k].append(i)
print results

Which outputs:

{'pycharm': [], 'i': [0, 1, 2, 3], 'stackoverflow': [1, 3], 'am': [0, 1, 2, 3], 'mr': [2], 'world': [0, 2], 'hello': [0, 1, 2, 3]}

The idea is using the h_unique list as keys in a dictionary (the results = {k: [] for k in h_unique} part).

Keys in dictionaries have the advantage of a constant lookup time, which is great for the if k in line: part (if it were a list, that in would take O(n) ) and then check if the word (the key k ) appears in the list. If it does, append the index of the list within the matrix to the dictionary of results.

Although I'm not certain this is what you want to achieve, though.

Answer 4

You can optimize your code to do the trick with

1. Using just a single for loop

2. Generators dictionaries for constant lookup time, as suggested previously. Generators are faster than for loops because the generate values on the fly

    In [75]: h_tokenized_doc = [ ['hello', 'world', 'i', 'am'], ...: ['hello', 'stackoverflow', 'i', 'am'], ...: ['hello', 'world', 'i', 'am', 'mr'], ...: ['hello', 'stackoverflow', 'i', 'am', 'pycahrm'] ] In [76]: h_unique = ('hello', 'world', 'i', 'am', 'stackoverflow', 'mr', 'pycharm') In [77]: term_id = {k: [] for k in h_unique} In [78]: for term in h_unique: ...: term_id[term].extend(i for i in range(len(h_tokenized_doc)) if term in h_tokenized_doc[i]) 

   which yields the output

    {'am': [0, 1, 2, 3], 'hello': [0, 1, 2, 3], 'i': [0, 1, 2, 3], 'mr': [2], 'pycharm': [], 'stackoverflow': [1, 3], 'world': [0, 2]} 

A more descriptive solution would be

In [79]: for term in h_unique:
    ...:     term_id[term].extend([(i,h_tokenized_doc[i].index(term)) for i in range(len(h_tokenized_doc)) if term in h_tokenized_doc[i]])


In [80]: term_id
Out[80]: 
{'am': [(0, 3), (1, 3), (2, 3), (3, 3)],
 'hello': [(0, 0), (1, 0), (2, 0), (3, 0)],
 'i': [(0, 2), (1, 2), (2, 2), (3, 2)],
 'mr': [(2, 4)],
 'pycharm': [],
 'stackoverflow': [(1, 1), (3, 1)],
 'world': [(0, 1), (2, 1)]}