Penalty Function
A Penalty Function is a measure function that ...
References
2010
- (Seni & Elder, 2010) ⇒ Giovanni Seni, John F. Elder. (2010). “Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions.” Morgan & Claypool. doi:10.2200/S00240ED1V01Y200912DMK002
- QUOTE: An influential paper was Tibshirani’s introduction of the Lasso regularization technique for linear models (Tibshirani, R., 1996). The Lasso uses the sum of the absolute value of the coefficients in the model as the penalty function and had roots in work done by Breiman on a coefficient post-processing technique which he had termed Garotte (Breiman et al., 1993). Another important development came with the LARS algorithm by Efron et al. (2004), which allows for an efficient iterative calculation of the Lasso solution. More recently, Friedman published a technique called Path Seeker (PS) that allows combining the Lasso penalty with a variety of loss (error) functions (Friedman and Popescu, 2004), extending the original Lasso paper which was limited to the Least-Squares loss. Careful comparison of the Lasso penalty with alternative penalty functions (e.g., using the sum of the squares of the coefficients) led to an understanding that the penalty function has two roles: controlling the “sparseness” of the solution (the number of coefficients that are non-zero) and controlling the magnitude of the non-zero coefficients (“shrinkage”). This led to development of the Elastic Net (Zou and Hastie, 2005) family of penalty functions which allow searching for the best shrinkage/sparseness tradeoff according to characteristics of the problem at hand (e.g., data size, number of input variables, correlation among these variables, etc.).