Statistical Model Assessment Task

Example(s):
- Predictive Model Assessment Task.
- …
Counter-Example(s):
- Model Deployment,
- Model Selection.
See: Cross-Validation, Statistical Inference, Algorithm Evaluation, Overfitting, ROC Analysis.

References

(Sammut & Webb, 2017) ⇒ (2017) "Model Evaluation". In: Sammut & Webb (2017).
- QUOTE: Model evaluation is the process of assessing a property or properties of a model.

(James et al., 2013) ⇒ Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. (2013). “An Introduction to Statistical Learning:with Applications in R.” Springer Publishing Company, Incorporated. ISBN:1461471370, 9781461471370
- QUOTE: The process of evaluating a model’s performance is known as model assessment, whereas model the process of selecting the proper level of flexibility for a model is known as assessment model selection. The bootstrap is used in several contexts, most commonly model to provide a measure of accuracy of a parameter estimate or of a given selection statistical learning method.

(Vehtari & Ojanen, 2012) ⇒ Aki Vehtari, and Janne Ojanen. (2012). “A Survey of Bayesian Predictive Methods for Model Assessment, Selection and Comparison.” In: Statistics Surveys Journal, 6. ISBN:1935-7516 doi:10.1214/12-SS102
- QUOTE: In this survey, the term predictive model assessment refers to evaluating the predictive performance of the actual belief model as the maximized expected utility. The definition holds regardless of whether the the main focus is in the estimation of the prediction performance or the prediction task is considered as a subcomponent of a more comprehensive decision problem. After satisfactory model criticism the model [math]\displaystyle{ M_∗ }[/math] can be considered to adequately represent the uncertainties involved in the prediction task, and the beliefs about the future observation can be described by the posterior predictive distribution [math]\displaystyle{ p(y|D, M_∗) }[/math]. Obtaining the optimal prediction [math]\displaystyle{ \hat{a} }[/math] and evaluating the maximized expected utility [math]\displaystyle{ u(M_∗, \hat{a}) }[/math], where the model label is now written explicitly, proceeds exactly as described in Section 3.1. Model assessment is presented as a stylized decision theoretic problem in Fig. 2.

(Hastie et al., 2009) ⇒ Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. (2009). “The Elements of Statistical Learning: Data Mining, Inference, and Prediction; 2nd edition.” Springer-Verlag. ISBN:0387848576
- QUOTE: It is important to note that there are in fact two separate goals that we might have in mind:
  - Model selection: estimating the performance of different models in order to choose the best one.
  - Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.