Regularization of Logistic Regression coefficients in MATLAB

Question

I'm trying to implement a Logistic Regression with regularization (either L1 or L2). The mnrfit() function does not implement regularization. Is there any built-in function that can do the regularization or do I have to roll my own regularization code? If so, are there any tutorials that I can look at? The papers I have been looking at are rather mathematically dense.

Answer 1

Liblinear was the standard we used.

http://www.csie.ntu.edu.tw/~cjlin/liblinear/

Answer 2

L1 as well as L2 regularization are very easy to implement.

L1 regularization works by subtracting a fixed amount of the absolute value of your weights after each training step. So with an L1 regularization coefficient of eg 0.01, your weights (1.0, -2.0, 3.0) would become (0.99, -1.99, 2.99).

L2 regularization works by subtracting a percentage of your weights. With a coefficient of 0.01, this means multiplying your weight vector by 1. - 0.01 = 0.99. The weights (1.0, -2.0, 3.0) would become (0.99, -1.98, 2.97). This is also known as weight decay .

As you can see, L1 regularization pulls small weights towards 0. L2 regularization on the other side has almost no effect on small weights but drastically reduces large weights.