Linear Least-Squares Regression System

AKA: Linear Least-Squares Optimizer/Estimator.
Context:
- It can range from being an Simple Linear Least-Squares Regression System to being a Multivariate Least-Squares Regression System, depending on the number of numeric predictors.
- It can range from being an Ordinary Linear Least-Squares Regression System to being a Regularized Linear Least-Squares Regression System.
- It can solve Linear Least-Squares Regression for Bounds on the Variables Tasks.
- It can solve Linear Least-Squares Regression Task with Non-Negativity Constraints Tasks.
Example(s):
- sklearn.linear_model[1], a GLM system that implements:
  - sklearn.linear_model.LinearRegression(), an OLS system.
- statsmodels.api[2], which implements:
  - statsmodels.api.OLS() an ordinary least squares for i.i.d. errors . Examples: [3]
  - statsmodels.api.GLS() a generalized least squares for arbitrary covariance. Examples: [4]
  - statsmodels.api.WLS() : A weighted least squares for heteroskedastic errors. Examples: [5]
- numpy.linalg.lstsq [6] a Numpy mode that solves the equation a x = b by computing a vector x that minimizes the Euclidean 2-norm || b - a x ||.
  Examples:

Input:	Output:
import numpy as np import matplotlib.pyplot as plt f = np.poly1d([5, 1]) x = np.linspace(0, 10, 30) y = f(x) + 6np.random.normal(size=len(x)) xn = np.linspace(0, 10, 200) a = np.vstack([x, np.ones(len(x))]).T cf=np.linalg.lstsq(a, y)[0] print 'Regression Coefficients', cf yn=cf[1]+cf[0]xn plt.plot(x,y,'bo',label='data') plt.plot(xn,yn,'g-',label='lsq') plt.xlabel('$x$',size=24) plt.ylabel('$y$',size=24) plt.legend() plt.show()	Regression Coefficients [ 4.95315091 0.02915413]

scipy.optimize.lsq_linear A Scipy mode that solves a linear least-squares problem with bounds on the variables.
scipy.optimize.nnls, a Scipy mode that solves a linear least squares with non-negativity constraint.
scipy.sparse.linalg.lsmr, a Scipy mode that is an iterative solver for least-squares problems.

References

(Scikit Learn, 2017) ⇒ http://scikit-learn.org/stable/modules/linear_model.html Retrieved: 2017-30-07.
- QUOTE: The following are a set of methods intended for regression in which the target value is expected to be a linear combination of the input variables. In mathematical notion, if [math]\displaystyle{ \hat{y} }[/math] is the predicted value.
  [math]\displaystyle{ \hat{y}(w, x) = w_0 + w_1 x_1 + \cdots + w_p x_p }[/math]
  Across the module, we designate the vector [math]\displaystyle{ w = (w_1,\cdots, w_p) }[/math] as coef_ and [math]\displaystyle{ w_0 }[/math] as intercept_.

(Scipy, 2017) ⇒ The Scipy community (2008-2009). “numpy.linalg.lstsq" https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.lstsq.html Last updated on Jun 10, 2017
- numpy.linalg.lstsq(a, b, rcond=-1)
  Return the least-squares solution to a linear matrix equation.
  Solves the equation a x = b by computing a vector x that minimizes the Euclidean 2-norm || b - a x ||^2. The equation may be under-, well-, or over- determined (i.e., the number of linearly independent rows of a can be less than, equal to, or greater than its number of linearly independent columns). If a is square and of full rank, then x (but for round-off error) is the “exact” solution of the equation (...)

(Scipy, 2017) ⇒ The Scipy community (2008-2009). “scipy.optimize.nnls" https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.nnls.html Last updated on Jun 10, 2017
- scipy.optimize.nnls(A, b)
  Solve argmin_x || Ax - b ||_2 for x>=0. This is a wrapper for a FORTRAN non-negative least squares solver (...)

(Scipy, 2017) ⇒ The Scipy community (2008-2009). “scipy.sparse.linalg.lsmr" https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.lsmr.html Last updated on Jun 10, 2017
- numpy.linalg.lstsq(a, b, rcond=-1)
  Return the least-squares solution to a linear matrix equation.
  Solves the equation a x = b by computing a vector x that minimizes the Euclidean 2-norm || b - a x ||^2. The equation may be under-, well-, or over- determined (i.e., the number of linearly independent rows of a can be less than, equal to, or greater than its number of linearly independent columns). If a is square and of full rank, then x (but for round-off error) is the “exact” solution of the equation.