如何优化预处理所有文本文档而不使用for循环在每次迭代中预处理单个文本文档？

Question

I want to optimize the below code so that it can process the 3000 text data efficiently and that data then will be fed to TFIDF Vectorizer and linkage() for clustering.

So far, I have read the excel using pandas and saved the dataframe into list variable. Then I iterated on the list by each text element in the list into tokens and then filtering out the stopwords from the element. The filtered element is stored into another variable and that variable is stored in the list. So at the end, I created a list of processed text elements(from list).

I think that optimization can be performed when a list is created and when stopwords are filtered out and also while saving the data into two different variables: documents_no_stopwords and processed_words.

It would be great if someone can help me on this or suggest me a direction to follow.

temp=0
df=pandas.read_excel('File.xlsx')

for text in df['text'].tolist():
    temp=temp+1
    preprocessing(text)
    print temp


def preprocessing(word):

    tokens = tokenizer.tokenize(word)

    processed_words = []
    for w in tokens:
        if w in stop_words:
            continue
        else:
    ## a new list is created with only the nouns in them for each text document
            processed_words.append(w)
    ## This step creates a list of text documents with only the nouns in them
    documents_no_stopwords.append(' '.join(processed_words))
    processed_words=[]

Answer 1

You need to first make set of stop words and use list comprehension to filter the tokens.

def preprocessing(txt):
    tokens = word_tokenize(txt)
    # print(tokens)
    stop_words = set(stopwords.words("english"))
    tokens = [i for i in tokens if i not in stop_words]

    return " ".join(tokens)

string = "Hey this is Sam. How are you?"
print(preprocessing(string))

Output:

'Hey Sam . How ?'

And rather than using a for loop, use df.apply like below:

df['text'] = df['text'].apply(preprocessing)

Why sets are preferred over list

There are duplicate entries in stopwords.words() If you check len(stopwords.words()) and len(set(stopwords.words())) the length of set is smaller by few hundreds. That's why set is preferred here.

Here's the difference between performance using list and set

x = stopwords.words('english')
y = set(stopwords.words('english'))

%timeit new = [i for i in tokens if i not in x]
# 10000 loops, best of 3: 120 µs per loop

%timeit old = [j for j in tokens if j not in y]
# 1000000 loops, best of 3: 1.16 µs per loop

And furthermore list-comprehension is faster than normal for-loop .