Predicting on new data using locally weighted regression (LOESS/LOWESS)

Question

How to fit a locally weighted regression in python so that it can be used to predict on new data?

There is statsmodels.nonparametric.smoothers_lowess.lowess , but it returns the estimates only for the original data set; so it seems to only do fit and predict together, rather than separately as I expected.

scikit-learn always has a fit method that allows the object to be used later on new data with predict ; but it doesn't implement lowess .

Answer 1

Lowess works great for predicting (when combined with interpolation)! I think the code is pretty straightforward-- let me know if you have any questions! Matplolib Figure

import matplotlib.pyplot as plt
%matplotlib inline
from scipy.interpolate import interp1d
import statsmodels.api as sm

# introduce some floats in our x-values
x = list(range(3, 33)) + [3.2, 6.2]
y = [1,2,1,2,1,1,3,4,5,4,5,6,5,6,7,8,9,10,11,11,12,11,11,10,12,11,11,10,9,8,2,13]

# lowess will return our "smoothed" data with a y value for at every x-value
lowess = sm.nonparametric.lowess(y, x, frac=.3)

# unpack the lowess smoothed points to their values
lowess_x = list(zip(*lowess))[0]
lowess_y = list(zip(*lowess))[1]

# run scipy's interpolation. There is also extrapolation I believe
f = interp1d(lowess_x, lowess_y, bounds_error=False)

xnew = [i/10. for i in range(400)]

# this this generate y values for our xvalues by our interpolator
# it will MISS values outsite of the x window (less than 3, greater than 33)
# There might be a better approach, but you can run a for loop
#and if the value is out of the range, use f(min(lowess_x)) or f(max(lowess_x))
ynew = f(xnew)


plt.plot(x, y, 'o')
plt.plot(lowess_x, lowess_y, '*')
plt.plot(xnew, ynew, '-')
plt.show()

Answer 2

I've created a module called moepy that provides an sklearn-like API for a LOWESS model (incl. fit/predict). This enables predictions to be made using the underlying local regression models, rather than the interpolation method described in the other answers. A minimalist example is shown below.

# Imports
import numpy as np
import matplotlib.pyplot as plt
from moepy import lowess

# Data generation
x = np.linspace(0, 5, num=150)
y = np.sin(x) + (np.random.normal(size=len(x)))/10

# Model fitting
lowess_model = lowess.Lowess()
lowess_model.fit(x, y)

# Model prediction
x_pred = np.linspace(0, 5, 26)
y_pred = lowess_model.predict(x_pred)

# Plotting
plt.plot(x_pred, y_pred, '--', label='LOWESS', color='k', zorder=3)
plt.scatter(x, y, label='Noisy Sin Wave', color='C1', s=5, zorder=1)
plt.legend(frameon=False)

A more detailed guide on how to use the model (as well as its confidence and prediction interval variants) can be found here .

Answer 3

Consider using Kernel Regression instead.

statmodels has an implementation .

If you have too many data points, why not use sk.learn's radiusNeighborRegression and specify a tricube weighting function?

Answer 4

It's not clear whether it's a good idea to have a dedicated LOESS object with separate fit/predict methods like what is commonly found in Scikit-Learn. By contrast, for neural networks, you could have an object which stores only a relatively small set of weights. The fit method would then optimize the "few" weights by using a very large training dataset. The predict method only needs the weights to make new predictions, and not the entire training set.

Predictions based on LOESS and nearest neighbors, on the other hand, need the entire training set to make new predictions. The only thing a fit method could do is store the training set in the object for later use. If x and y are the training data, and x0 are the points at which to make new predictions, this object-oriented fit/predict solution would look something like the following:

model = Loess()
model.fit(x, y)         # No calculations. Just store x and y in model.
y0 = model.predict(x0)  # Uses x and y just stored.

By comparison, in my localreg library, I opted for simplicity:

y0 = localreg(x, y, x0)

It really comes down to design choices, as the performance would be the same. One advantage of the fit/predict approach is that you could have a unified interface like they do in Scikit-Learn, where one model could easily be swapped by another. The fit/predict approach also encourages a machine learning way to think of it, but in that sense LOESS is not very efficient, since it requires storing and using all the data for every new prediction. The latter approach leans more towards the origins of LOESS as a scatterplot smoothing algorithm, which is how I prefer to think about it. This might also shed some light on why statsmodel do it the way they do.

Answer 5

Check out the loess class in scikit-misc . The fitted object has a predict method:

loess_fit = loess(x, y, span=.01);
loess_fit.fit();
preds = loess_fit.predict(x_new).values

https://has2k1.github.io/scikit-misc/stable/generated/skmisc.loess.loess.html

Answer 6

How to fit a locally weighted regression in python so that it can be used to predict on new data?

There is statsmodels.nonparametric.smoothers_lowess.lowess , but it returns the estimates only for the original data set; so it seems to only do fit and predict together, rather than separately as I expected.

scikit-learn always has a fit method that allows the object to be used later on new data with predict ; but it doesn't implement lowess .