[英]Scikit-Learn GridSearchCV failing on on a gensim LDA model
[英]Reproducible LDA Model in Scikit-learn
我正在使用LDA进行主题建模。
从sklearn.decomposition导入LatentDirichletAllocation
我使用一组10个文件制作了模型。 现在,我尝试将其群集为3。
类似于以下内容:
“””
import numpy as np
data = []
a1 = " a word in groupa doca"
a2 = " a word in groupa docb"
a3 = "a word in groupb docc"
a4 = "a word in groupc docd"
a5 ="a word in groupc doce"
data = [a1,a2,a3,a4,a5]
del a1,a2,a3,a4,a5
NO_DOCUMENTS = len(data)
print(NO_DOCUMENTS)
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
NUM_TOPICS = 2
vectorizer = CountVectorizer(min_df=0.001, max_df=0.99998,
stop_words='english', lowercase=True,
token_pattern='[a-zA-Z\-][a-zA-Z\-]{2,}')
data_vectorized = vectorizer.fit_transform(data)
# Build a Latent Dirichlet Allocation Model
lda_model = LatentDirichletAllocation(n_topics=NUM_TOPICS,
max_iter=10, learning_method='online')
lda_Z = lda_model.fit_transform(data_vectorized)
vocab = vectorizer.get_feature_names()
text = "The economy is working better than ever"
x = lda_model.transform(vectorizer.transform([text]))[0]
print(x, x.sum())
for iDocIndex,text in enumerate(data):
x = list(lda_model.transform(vectorizer.transform([text]))[0])
maxIndex = x.index(max(x))
if TOPICWISEDOCUMENTS[maxIndex]:
TOPICWISEDOCUMENTS[maxIndex].append(iDocIndex)
else:
TOPICWISEDOCUMENTS[maxIndex] = [iDocIndex]
print(TOPICWISEDOCUMENTS)
“””
每当我运行系统时,即使对于同一组输入数据,我也会获得不同的集群。
或者,LDA是不可复制的。
如何使其可再现..?
为了在scikit中重现性,请在代码中看到的任何位置设置random_state
参数。
在您的情况下,其LatentDirichletAllocation(...)
用这个:
lda_model = LatentDirichletAllocation(n_topics=NUM_TOPICS,
max_iter=10,
learning_method='online'
random_state = 42)
检查此链接:
如果要使整个脚本具有可复制性,并且不想搜索将random_state
放在random_state
,则可以设置一个全局numpy随机种子。
import numpy as np
np.random.seed(42)
请参阅: http : //scikit-learn.org/stable/faq.html#how-do-i-set-a-random-state-for-an-entire-execution
lda_model = LatentDirichletAllocation(n_topics=NUM_TOPICS,
max_iter=10,
learning_method='online'
random_state = 42)
工作... !!!
非常感谢
另外,我也尝试过
import numpy as np
np.random.seed(42)
但这是无效的。
感谢您的解决
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.