CountVectorizer fit_transform 花费的时间太长

Question

def tokenize(text):
    text = re.sub('[^a-zA-Z0-9]', ' ', text)
    words = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()
    words = [lemmatizer.lemmatize(w.lower().strip()) for w in words if w not in stopwords.words()]
    return words

pipeline = Pipeline([
('vect', CountVectorizer(tokenizer=tokenize)),
#     ('tfidf', TfidfTransformer()),
#     ('clf', MultiOutputClassifier(RandomForestClassifier()))
])

Given above code, CountVectorizer takes too long (ran for 60 minutes but it did not finished) to fit but if I remove line if w not in stopwords.words() it just take 5 minutes to fit, what could be problem and possible solution with this code. I am using stop words from nltk.corpus.

Note: tokenize function works fine, using separately for any text input.

Thank you

Answer 1

My first guess is that the function stopwords.words() does some heavy job on each call. Maybe, you could try caching it. The same is true for lemmatizer: calling the constructor only once can speed up the code significantly.

stop_set = set(stopwords.words())
lemmatizer = WordNetLemmatizer()

def tokenize(text):
    text = re.sub('[^a-zA-Z0-9]', ' ', text)
    words = word_tokenize(text)
    words = [lemmatizer.lemmatize(w.lower().strip()) for w in words if w not in stop_set]
    return words 

in my experience, it can help to cache even the lemmatization function, like

from functools import lru_cache

stop_set = set(stopwords.words())
lemmatizer = WordNetLemmatizer()

@lru_cache(maxsize=10000)
def lemmatize(word):
    return lemmatizer.lemmatize(w.lower().strip())

def tokenize(text):
    text = re.sub('[^a-zA-Z0-9]+', ' ', text)
    words = [lemmatize(w) for w in word_tokenize(text)]
    return [w for w in words if w not in stop_set]