Regularized Objective Function

A Regularized Objective Function is an objective function that includes a regularization term.

• AKA: Regularization Objective Function , Penalized Fitting Objective Function.
• Context:on
  • It can be is defined as

    [math]\displaystyle{ E_{reg}=E(f)+\lambda R(f) }[/math]

    where is [math]\displaystyle{ E(f) }[/math] is unregularized objective function, [math]\displaystyle{ \lambda }[/math] is the regularization term and [math]\displaystyle{ R(f) }[/math] is the regularization function.

• Example(s):
  • A ridge regression regularized objective function [math]\displaystyle{ E(f)=\sum_{i=1}^n \sum_{j=0}^p \parallel y_i - x_{ij}\beta_j\parallel^2 + \lambda \sum_{j=0}^p \parallel\beta_j\parallel^2 }[/math]
  • A lasso regression regularized objective function [math]\displaystyle{ E(f)=\sum_{i=1}^n \sum_{j=0}^p \parallel y_i - x_{ij}\beta_j\parallel^2 + \lambda \sum_{j=0}^p \parallel\beta_j\parallel }[/math]
• Counter-Example(s):
  • A linear least-squares regression objective function [math]\displaystyle{ E(f)=\sum_{i=1}^n \sum_{j=0}^p \parallel y_i - x_{ij}\beta_j\parallel^2 }[/math]
  • A Maximum Likelihood Estimation objective function [math]\displaystyle{ E(f)= - \ln\mathcal{L}(\boldsymbol{\beta}\,;\,x_1,\ldots,x_n) = - \sum_{i=1}^n \ln f(x_i\mid\boldsymbol{\beta}) }[/math]
• See: Regularization, Loss Function, Least-Squares Regression Task.

---

---

References

2017a

• (Quadianto & Buntine, 2017) ⇒ Quadrianto, N., & Buntine, W. L. (2017). "Regression". In "Encyclopedia of Machine Learning and Data Mining" Sammut, C., & Webb, G. I. (Eds.). (2017) pp 1075-1080. Springer US.
  • QUOTE: Fitting

    In general, regression fits a model to data using an objective function or quality criterion in a form such as

    [math]\displaystyle{ \begin{array}{rcl} E(f)\ =\ \sum _{i=1}^{N}\epsilon (y_{ i},f(x_{i}))\, & & {}\\ \end{array} }[/math]

    where smaller E(f) implies better quality. This might be derived as an error/loss function or as a negative log likelihoodor log probability. The squared error function is the most convenient (leading to a least squares calculation), but many possibilities exist. In general, methods are distinguished by three aspects: (1) the representation of the function f(), (2) the form of the term [math]\displaystyle{ \epsilon (y_{i},f(x_{i}) }[/math], and (3) the penalty term discussed next.

    Regularized/Penalized Fitting

    The issue of overfitting, as mentioned already in the section Motivation and Background, is usually addressed by introducing a regularization or penalty term to the objective function. The regularized objective function is now in the form of:

    [math]\displaystyle{ \begin{array}{rcl} E_{\mathrm{reg}} = E(f) +\lambda R(f).& &{}\end{array}\quad }[/math](1)

    Here, E(f) measures the quality of the solution for f() on the observed data points, R(f) penalizes complexity of f(), and λ is called the regularization parameter which controls the relative importance between the two. Measures of function curvature, for instance, can be used for R(f). In standard support vector machines, the term E(f) measures the hinge loss, and penalty R(f) is the sum of squares of the parameters, also used in ridge regression (Hastie et al., 2003).

---

2017

• (Wikipedia, 2017) ⇒ https://en.wikipedia.org/wiki/Loss_function Retrieved:2017-8-27.
  • In mathematical optimization, statistics, econometrics, decision theory, machine learning and computational neuroscience, a loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost" associated with the event. An optimization problem seeks to minimize a loss function. An objective function is either a loss function or its negative (in specific domains, also variously called a reward function, a profit function, a utility function, a fitness function, etc.), in which case it is to be maximized.

    In statistics, typically a loss function is used for parameter estimation, and the event in question is some function of the difference between estimated and true values for an instance of data. The concept, as old as Laplace, was reintroduced in statistics by Abraham Wald in the middle of the 20th century. In the context of economics, for example, this is usually economic cost or regret. In classification, it is the penalty for an incorrect classification of an example. In actuarial science, it is used in an insurance context to model benefits paid over premiums, particularly since the works of Harald Cramér in the 1920s. In optimal control the loss is the penalty for failing to achieve a desired value. In financial risk management the function is mapped to a monetary loss.