L2-Norm Regularized Optimization Task

An L2-Norm Regularized Optimization Task is a regularized optimization task that is a function optimization task which includes an L2-norm regularization term in its objective function.

• AKA: ℓ2 Regularization, Ridge Regularization, Tikhonov Regularization, Weight Decay.
• Context:
  • It can (typically) minimize an objective function that combines an empirical loss term with an L2-norm regularization term.
  • It can (typically) penalize large model parameters by adding the squared L2-norm of the weight vector scaled by a regularization coefficient.
  • It can (typically) solve the following Unconstrained Optimization Task [math]\displaystyle{ \min_{\mathbf{w}} \tfrac{1}{2} \mathbf{w}^T \mathbf{w} + C \sum_{i=1}^l \log (1 + e^{(-y_i \mathbf{w}^T \mathbf{x}_i)}) }[/math], where [math]\displaystyle{ \mathbf{x} }[/math] is a labeled training record and [math]\displaystyle{ l }[/math] is the training set size.
  • It can (typically) shrink model coefficients toward zero without forcing them to exactly zero, producing a dense solution vector.
  • It can (typically) improve model generalization by reducing model overfitting on the training dataset.
  • It can (typically) trade off training error against model complexity via its regularization coefficient.
  • It can (typically) produce a strongly convex objective function when the underlying loss function is convex, guaranteeing a unique global optimum.
  • It can (typically) be solved by gradient-based optimization methods because the L2-norm penalty is differentiable everywhere.
  • It can (typically) stabilize ill-posed problems and improve the numerical conditioning of a linear system.
  • It can (often) be interpreted as a maximum a posteriori estimation task with a Gaussian prior on the model parameters.
  • It can (often) reduce the variance of a statistical estimator at the cost of introducing estimation bias.
  • It can (often) mitigate multicollinearity among input features in a regression model.
  • It can (often) be combined with an L1-norm regularization term to form an elastic net regularization task.
  • It can (often) require hyperparameter tuning of its regularization coefficient via cross-validation.
  • It can (often) be applied to deep neural networks as weight decay during stochastic gradient descent.
  • It can range from being a Linear L2-Norm Regularized Optimization Task to being a Nonlinear L2-Norm Regularized Optimization Task, depending on its model class.
  • It can range from being a Constrained L2-Norm Regularized Optimization Task to being an Unconstrained L2-Norm Regularized Optimization Task, depending on its optimization formulation.
  • It can range from being a Weakly Regularized Optimization Task to being a Strongly Regularized Optimization Task, depending on its regularization coefficient value.
  • …
• Example(s):
  • Regularized Regression Tasks, such as:
    • Ridge Regression Task for linear regression with an L2-norm penalty.
    • L2 Regularized Logistic Regression Task for binary classification.
  • Regularized Support Vector Machine Tasks, such as:
    • L2-Regularized L2-Loss Support Vector Classification Task as implemented in LIBLINEAR (2008).
    • L2-Regularized Logistic Regression Task as implemented in LIBLINEAR (2008).
  • Regularized Neural Network Training Tasks, such as:
    • Weight Decay Regularization as described by Hanson & Pratt (1989).
    • AdamW Optimization Task with decoupled weight decay as described by Loshchilov & Hutter (2017).
  • Kernel Method regularization, such as:
    • Kernel Ridge Regression Task using an L2-norm penalty in a reproducing kernel Hilbert space.
  • …
• Counter-Example(s):
  • L1 Norm Regularization, which uses an L1-norm penalty to produce a sparse solution vector.
  • L1-Norm Regularized Optimization Task, which minimizes an L1-norm penalty over the weight vector.
  • L0-Norm Regularized Optimization Task, which penalizes the number of nonzero parameters directly.
  • Unregularized Optimization Task, which lacks any regularization term in its objective function.
  • …
• See: L2 Regularized Logistic Regression, L2-Norm Regularizer, Regularization, L2-norm, Regularization Coefficient, Ridge Regression, Tikhonov Regularization, Convex Optimization Task, Overfitting, Weight Decay.

---

References

• • Fan, Rong-En; Chang, Kai-Wei; Hsieh, Cho-Jui; Wang, Xiang-Rui; Lin, Chih-Jen (2008). "LIBLINEAR: A Library for Large Linear Classification". In: Journal of Machine Learning Research, 9:1871-1874.
  • Loshchilov, Ilya; Hutter, Frank (2017). "Decoupled Weight Decay Regularization". arXiv:1711.05101.
  • Hoerl, Arthur E.; Kennard, Robert W. (1970). "Ridge Regression: Biased Estimation for Nonorthogonal Problems". In: Technometrics, 12(1):55-67.