# Least-Squares Function Fitting Algorithm

(Redirected from method of least squares)

A Least-Squares Function Fitting Algorithm is a function fitting algorithm can be implemented by a least-squares system to solve a least-squares task (to provide a minimizing least-squares solution).

## References

### 2014a

• (Wikipedia, 2014) ⇒ http://en.wikipedia.org/wiki/least_squares Retrieved:2014-11-23.
• The method of least squares is a standard approach to the approximate solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the errors made in the results of every single equation.

The most important application is in data fitting. The best fit in the least-squares sense minimizes the sum of squared residuals, a residual being the difference between an observed value and the fitted value provided by a model. When the problem has substantial uncertainties in the independent variable (the 'x' variable), then simple regression and least squares methods have problems; in such cases, the methodology required for fitting errors-in-variables models may be considered instead of that for least squares.

Least squares problems fall into two categories: linear or ordinary least squares and non-linear least squares, depending on whether or not the residuals are linear in all unknowns. The linear least-squares problem occurs in statistical regression analysis; it has a closed-form solution. A closed-form solution (or closed-form expression) is any formula that can be evaluated in a finite number of standard operations. The non-linear problem has no closed-form solution and is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases.

When the observations come from an exponential family and mild conditions are satisfied, least-squares estimates and maximum-likelihood estimates are identical. The method of least squares can also be derived as a method of moments estimator. The following discussion is mostly presented in terms of linear functions but the use of least-squares is valid and practical for more general families of functions. Also, by iteratively applying local quadratic approximation to the likelihood (through the Fisher information), the least-squares method may be used to fit a generalized linear model. For the topic of approximating a function by a sum of others using an objective function based on squared distances, see least squares (function approximation). The least-squares method is usually credited to Carl Friedrich Gauss (1795),[1] but it was first published by Adrien-Marie Legendre.

1. Cite error: Invalid <ref> tag; no text was provided for refs named brertscher

### 2014b

• http://en.wikipedia.org/wiki/Least_squares#Problem_statement
• The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of n points (data pairs) $(x_i,y_i)\!$, i = 1, ..., n, where $x_i\!$ is an independent variable and $y_i\!$ is a dependent variable whose value is found by observation. The model function has the form $f(x,\beta)$, where the m adjustable parameters are held in the vector $\boldsymbol \beta$. The goal is to find the parameter values for the model which "best" fits the data. The least squares method finds its optimum when the sum, S, of squared residuals :$S=\sum_{i=1}^{n}{r_i}^2$ is a minimum. A residual is defined as the difference between the actual value of the dependent variable and the value predicted by the model. :$r_i=y_i-f(x_i,\boldsymbol \beta)$.

### 2012

Solver F(x) Constraints
\ C·x – d None
lsqnonneg C·x – d x ≥ 0
lsqlin C·x – d Bound, linear
lsqnonlin General F(x) Bound
lsqcurvefit F(x, xdata) – ydata Bound

### 1805

• (Legendre, 1805) ⇒ Adrien-Marie Legendre. (1805). “Nouvelle formula pour réduire en distances vraies les distances apparentes de la Lune au Soleil ou à une étoile."
• https://archive.org/stream/sourcebookinmath00smit#page/576/mode/2up
• QUOTE: … Of all the principles which can be proposed for [making estimates from a sample], I think there is none more general, more exact, and more easy of application, than that of which we have made use… which consists of rendering the sum of the squares of the errors a minimum. ...