如何使用机器学习模型来预测特征略有不同的数据？

Question

I have a randomForest model trained on a bunch of NLP data (tf-idf values for each word). I want to use it to predict on a new dataset. The features in the model overlap with but don't quite match the features in the new data, such that when I predict on the new data I get:

Error in predict.randomForest(object = model, newdata = new_data) : 
  variables in the training data missing in newdata

I thought to get around this error by excluding all the features from the model which do not appear in the new data, and all the features in the new data which do not appear in the model. Putting aside for the moment the impact on model accuracy (this would significantly pare down the number of features, but there would still be plenty to predict with), I did something like this:

model$forest$xlevels <- model$forest$xlevels[colnames(new_data)]
# and vice versa
new_data <- new_data[names(model$forest$xlevels)]

This worked, insofar as names(model$forest$xlevels) == colnames(new_data) returned TRUE for each feature name.

However, when I try to predict on the resulting new_data I still get the variables in the training data missing in newdata error. I am fairly certain that I'm amending the correct part of the model ( model$forest$xlevels ), so why isn't it working?

Answer 1

i think you should go the other way around. That is add the missing columns to the newdata.

When you are working with bags of words, it is common to have words that are not present in some batch of new data. These missing words should just be encoded as a columns of zeros.

# do something like this (also exclude the target variable, obviously)
names_missing <- names(traindata)[!names(traindata) %in% names(new_data)]
new_data[,names_missing] <- 0L

and then you should be able to predict