在scikit-learn中拟合分类器之前进行特征缩放的必要性

Question

I used to believe that scikit-learn 's Logistic Regression classifier (as well as SVM ) automatically standardizes my data before training. The reason I used to believe it is because of the regularization parameter C that is passed to the LogisticRegression constructor: Applying regularization (as I understand it) doesn't make sense without feature scaling. For regularization to work properly, all the features should be on comparable scales. Therefore, I used to assume that when calling the LogisticRegression.fit(X) on training data X , the fit method first performs feature scaling and then starts training. In order to test my assumption I've decided to manually scale the features of X as follows:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X)
X_std = scaler.transform(X)

Then I've initialized a LogisticRegression object with a regularization parameter C :

from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression(C=10.0, random_state=0)

I've found out that training the model on X is not equivalent to training the model on X_std . That is to say, the model produced by

log_reg.fit(X_std, y)

is not similar to the model produced by

log_reg.fit(X, y)

Does that mean that scikit-learn doesn't standardize the features before training? Or maybe it does scale but by applying a different procedure? If scikit-learn doesn't perform feature scaling, how is it consistent with requiring the regularization parameter C ? Should I manually standardize my data every time before fitting the model in order for regularization to make sense?

Answer 1

From the following note in: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

I'd assume that you need to preprocess the data yourself (eg with a scaler from sklearn.preprocessing.)

solver : {'newton-cg', 'lbfgs', 'liblinear', 'sag'}

Algorithm to use in the optimization problem. For small datasets, 'liblinear' is a good choice, whereas 'sag' is faster for large ones.

For multiclass problems, only 'newton-cg' and 'lbfgs' handle multinomial loss; 'sag' and 'liblinear' are limited to one-versus-rest schemes.

'newton-cg', 'lbfgs' and 'sag' only handle L2 penalty.

Note that 'sag' fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from sklearn.preprocessing.