Locally weighted linear Regression using Python
Locally weighted linear regression is the nonparametric regression methods that combine k-nearest neighbor based machine learning. It is referred to as locally weighted because for a query point the function is approximated on the basis of data near that and weighted because the contribution is weighted by its distance from the query point.
Locally Weighted Regression (LWR) is a non-parametric, memory-based algorithm, which means it explicitly retains training data and used it for every time a prediction is made.
To explain the locally weighted linear regression, we first need to understand the linear regression. The linear regression can be explained with the following equations:
Let (xi, yi) be the query point, then for minimizing the cost function in the linear regression:
by calculating so, that it minimize the above cost function. Our output will be:
Thus, the formula for calculating \theta can also be:
where, beta is the vector of linear vector, X, Y is the matrix, and vector of all observations.
For locally weighted linear regression:
by calculating so, that it minimize the above cost function. Our output will be:
Here, w(i) is the weight associated with each observation of training data. It can be calculated by the given formula:
Or this can be represented in the form of a matrix calculation:
where x(i) is the observation from the training data and x is a particular point from which the distance is calculated and T(tau) is the bandwidth. Here, T(tau) decides the amount of fitness in the function, if the function is closely fitted, its value will be small. Therefore,
then, we can calculate \theta with the following equation:
Implementation
- For this implementation, we will be using bokeh. If you want to know bokeh functionalities in details please check this article
Python3
# Necessary imports import numpy as np from ipywidgets import interact from bokeh.plotting import figure, show, output_notebook from bokeh.layouts import gridplot % matplotlib inline output_notebook() plt.style.use( 'seaborn-dark' ) # function to perform locally weighted linear regression def local_weighted_regression(x0, X, Y, tau): # add bias term x0 = np.r_[ 1 , x0] X = np.c_[np.ones( len (X)), X] # fit model: normal equations with kernel xw = X.T * weights_calculate(x0, X, tau) theta = np.linalg.pinv(xw @ X) @ xw @ Y # "@" is used to # predict value return x0 @ theta # function to perform weight calculation def weights_calculate(x0, X, tau): return np.exp(np. sum ((X - x0) * * 2 , axis = 1 ) / ( - 2 * (tau * * 2 ) )) # plot locally weighted regression for different bandwidth values def plot_lwr(tau): # prediction domain = np.linspace( - 3 , 3 , num = 300 ) prediction = [local_regression(x0, X, Y, tau) for x0 in domain] plot = figure(plot_width = 400 , plot_height = 400 ) plot.title.text = 'tau=%g' % tau plot.scatter(X, Y, alpha = . 3 ) plot.line(domain, prediction, line_width = 2 , color = 'red' ) return plot #define distribution n = 1000 # generate dataset X = np.linspace( - 3 , 3 , num = n) Y = np. abs (X * * 3 - 1 ) # jitter X X + = np.random.normal(scale = . 1 , size = n) # show the plots for different values of Tau show(gridplot([ [plot_lwr( 10. ), plot_lwr( 1. )], [plot_lwr( 0.1 ), plot_lwr( 0.01 )] ])) |
- As we can notice from the above plot that with small values of bandwidth, the model fits better but sometimes it will lead to overfitting.
References:
- Stanford CS 229