我如何在机器学习中使用不同的数据集测试我的 model

Question

im new in machine learning and i am create a one small project using CountVectorizer model. i am split my data to 80% -20%. 80% for training model and 20% for testing model. my model work properly run on 20% test data but can i used to test my model on different data set that is similar to training data set?

i am using joblib for dump and load my model.

from joblib import dump, load
dump(pipe, filename)

loaded_model = load('filename')

my question is how i directly test my model using different dataset?

Answer 1

Yes, you can use the model to test similar datasets.

However, you must keep in mind the preprocessing step according to the model.

When you trained your model, it was trained on a particular dimension and the size of input would have been AxB matric. When you have a new test sentence or new dataset, you must first do the same preprocessing, otherwise, it will throw dimension mismatch errors.

Example:

suppose you have the following count vectorizer object

cv = CountVectorizer()

then you must first fit it on your training dataset, for say

X = dataframe['text_column_name']
X = cv.fit_transform(X) # Fit the Data

Once this is done, whenever you have a new sentence, say

test_sentence = "this is a test sentence"

then you must use the cv object in the following manner

model_input = cv.transform([test_sentence]).toarray()

and then you can make predictions:

model.predict(model_input)

This method must be followed even if you wish to test a new dataset which is in a data frame or some other file format.