使用泡菜保存 model

Question

我已經建立了一個分類器，我想保存它以備將來使用。 分類器包括不同的算法（邏輯回歸、朴素貝葉斯、支持向量機）：

X, y = tfidf(df, ngrams = 1)
X, y = under_sample.fit_resample(X, y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=40)
df_result = df_result.append(training_naive(X_train, X_test, y_train, y_test), ignore_index = True)
df_result = df_result.append(training_logreg(X_train, X_test, y_train, y_test), ignore_index = True)
df_result = df_result.append(training_svm(X_train, X_test, y_train, y_test), ignore_index = True)

這是我代碼的最后一步，我比較不同的算法。 training_svm/logreg 和 naive 是函數。 例如 training_svm 定義如下：

def training_svm(X_train_log, X_test_log, y_train_log, y_test_log):
    
    folds = StratifiedKFold(n_splits = 3, shuffle = True, random_state = 40)
    
    clf = svm.SVC(kernel='linear') # Linear Kernel
    
    clf.fit(X_train_log, y_train_log)

    res = pd.DataFrame(columns = ['Preprocessing', 'Model', 'Precision', 'Recall', 'F1-score', 'Accuracy'])
    
    y_pred = clf.predict(X_test_log)
    
    f1 = f1_score(y_pred, y_test_log, average = 'weighted')
    pres = precision_score(y_pred, y_test_log, average = 'weighted')
    rec = recall_score(y_pred, y_test_log, average = 'weighted')
    acc = accuracy_score(y_pred, y_test_log)
    
    res = res.append({'Model': f'SVM', 'Precision': pres, 
                     'Recall': rec, 'F1-score': f1, 'Accuracy': acc}, ignore_index = True)

    return res

由於我想使用新數據並對其進行測試，所以我想知道如何保存並重新使用它。 我會說我應該做這樣的事情

import pickle

# save
with open('model.pkl','wb') as f:
    pickle.dump(clf,f)

# load
with open('model.pkl', 'rb') as f:
    clf2 = pickle.load(f)

clf2.predict(X[0:1])

你能解釋一下如何將它擴展到我的項目嗎？

Answer 1

正如sklearn所說：

通過使用Python內置的持久化model，即pickle，可以在scikit-learn中保存一個model，即pickle

例子：

from sklearn import svm
from sklearn import datasets
clf = svm.SVC()
X, y= datasets.load_iris(return_X_y=True)
clf.fit(X, y)

import pickle
s = pickle.dumps(clf)
clf2 = pickle.loads(s)
clf2.predict(X[0:1])

然后，您可以將它包含在每個 model 的代碼中，從而將 function 稱為

def predict_svm(to_predict):
    with open("'your_svm_model'",'rb') as f_input:
        clf = pickle.loads(f_input) # maybe handled with a singleton to reduce loading for multiple predictions
    return clf.predict(to_predict)

無論如何，sklearn 建議使用joblib ：

In the specific case of scikit-learn, it may be better to use joblib's replacement of pickle (dump & load), which is more efficient on objects that carry large numpy arrays internally as is often the case for fitted scikit-learn estimators, but can only pickle to磁盤而不是字符串：

from joblib import dump, load
dump(clf, 'filename.joblib') 

clf = load('filename.joblib') 

詳情在這里