[英]How do I store a TfidfVectorizer for future use in scikit-learn?
我有一個TfidfVectorizer
,它可以對文章集合進行矢量化,然后進行特征選擇。
vectroizer = TfidfVectorizer()
X_train = vectroizer.fit_transform(corpus)
selector = SelectKBest(chi2, k = 5000 )
X_train_sel = selector.fit_transform(X_train, y_train)
現在,我想存儲它並在其他程序中使用它。 我不想在訓練數據集上重新運行TfidfVectorizer()
和特征選擇器。 我怎么做? 我知道如何使用joblib
使模型持久化,但我想知道這是否與使模型持久化相同。
您可以簡單地使用內置的pickle庫:
import pickle
pickle.dump(vectorizer, open("vectorizer.pickle", "wb"))
pickle.dump(selector, open("selector.pickle", "wb"))
並加載它:
vectorizer = pickle.load(open("vectorizer.pickle", "rb"))
selector = pickle.load(open("selector.pickle", "rb"))
Pickle 會將對象序列化到磁盤並在您需要時再次將它們加載到內存中
這是我使用 joblib 的回答:
import joblib
joblib.dump(vectorizer, 'vectorizer.pkl')
joblib.dump(selector, 'selector.pkl')
稍后,我可以加載它並准備好:
vectorizer = joblib.load('vectorizer.pkl')
selector = joblib.load('selector.pkl')
test = selector.trasnform(vectorizer.transform(['this is test']))
“使對象持久化”基本上意味着您將轉儲存儲在內存中的二進制代碼,該代碼代表硬盤驅動器上的文件中的對象,以便稍后在您的程序或任何其他程序中該對象可以從硬盤驅動器中的文件重新加載到內存中。
scikit-learn 包含joblib
或 stdlib pickle
和cPickle
都可以完成這項工作。 我傾向於更喜歡cPickle
因為它明顯更快。 使用ipython 的 %timeit 命令:
>>> from sklearn.feature_extraction.text import TfidfVectorizer as TFIDF
>>> t = TFIDF()
>>> t.fit_transform(['hello world'], ['this is a test'])
# generic serializer - deserializer test
>>> def dump_load_test(tfidf, serializer):
...: with open('vectorizer.bin', 'w') as f:
...: serializer.dump(tfidf, f)
...: with open('vectorizer.bin', 'r') as f:
...: return serializer.load(f)
# joblib has a slightly different interface
>>> def joblib_test(tfidf):
...: joblib.dump(tfidf, 'tfidf.bin')
...: return joblib.load('tfidf.bin')
# Now, time it!
>>> %timeit joblib_test(t)
100 loops, best of 3: 3.09 ms per loop
>>> %timeit dump_load_test(t, pickle)
100 loops, best of 3: 2.16 ms per loop
>>> %timeit dump_load_test(t, cPickle)
1000 loops, best of 3: 879 µs per loop
現在,如果您想在單個文件中存儲多個對象,您可以輕松創建一個數據結構來存儲它們,然后轉儲數據結構本身。 這將適用於tuple
、 list
或dict
。 從你的問題的例子:
# train
vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(corpus)
selector = SelectKBest(chi2, k = 5000 )
X_train_sel = selector.fit_transform(X_train, y_train)
# dump as a dict
data_struct = {'vectorizer': vectorizer, 'selector': selector}
# use the 'with' keyword to automatically close the file after the dump
with open('storage.bin', 'wb') as f:
cPickle.dump(data_struct, f)
稍后或在另一個程序中,以下語句將帶回程序內存中的數據結構:
# reload
with open('storage.bin', 'rb') as f:
data_struct = cPickle.load(f)
vectorizer, selector = data_struct['vectorizer'], data_struct['selector']
# do stuff...
vectors = vectorizer.transform(...)
vec_sel = selector.transform(vectors)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.