从 gensim 导入 Lemmatization 的问题

Question

I am trying to use Gensim packages as written below:

import re, numpy as np, pandas as pd
from pprint import pprint

# Gensim
import gensim, spacy, logging, warnings
import gensim.corpora as corpora
from gensim.utils import lemmatize, simple_preprocess
from gensim.models import CoherenceModel
import matplotlib.pyplot as plt

But i keep getting the error:

ImportError: cannot import name 'lemmatize' from 'gensim.utils' (/Users/xxx/opt/anaconda3/envs/virt_env/lib/python3.9/site-packages/gensim/utils.py)

I am using gensim v4.0.1, Python 3.8, numpy 1.20.0.

Has anyone encountered this kinda problem lately? Thank you

Answer 1

Gensim only ever previously wrapped the lemmatization routines of another library ( Pattern ) – which was not a particularly modern/maintained option, so was removed from Gensim-4.0.

Users should choose & apply their own lemmatization operations, if any, as a preprocessing step before applying Gensim's algorithms. Some Python libraries offering lemmatization include:

• Pattern (Gensim's previously-included option): https://github.com/clips/pattern
• NLTK: https://www.nltk.org/api/nltk.stem.html#nltk.stem.wordnet.WordNetLemmatizer
• UDPipe: https://ufal.mff.cuni.cz/udpipe
• Spacy: https://spacy.io/api/lemmatizer
• Stanza: https://stanfordnlp.github.io/stanza/