Classification Accuracy Metric
(Redirected from Accuracy Function)
Jump to navigation
Jump to search
A Classification Accuracy Metric is a categorical prediction task performance measure that assesses the proportion of correct classifications made by a classification system on labeled test data.
- AKA: Accuracy Measure, Classification Accuracy, Overall Accuracy, Global Accuracy, Prediction Accuracy, Model Accuracy.
- Context:
- It can (typically) be calculated as (True Positives + True Negatives) / (True Positives + True Negatives + False Positives + False Negatives) for binary classification tasks.
- It can (typically) be calculated as the number of Correct Classifications divided by the total number of Classification Acts for multi-class classification tasks.
- It can (typically) be the inverse measure of a Classification Error Rate.
- It can (typically) range from 0.0 (all predictions wrong) to 1.0 (all predictions correct), often expressed as a percentage.
- It can (often) be misleading on Imbalanced Datasets where one class dominates.
- It can (often) be estimated using Cross-Validation, Bootstrap Sampling, or Hold-Out Validation.
- It can (often) require that Test Cases be unseen during the Training Phase to avoid Overfitting Bias.
- It can (often) be reported alongside other metrics like Precision, Recall, and F1-Score for comprehensive evaluation.
- It can (often) fail to capture Class-Specific Performance in multi-class problems.
- It can range from being a Binary Classification Accuracy to being a Multi-Class Classification Accuracy, depending on its class count.
- It can range from being a Balanced Accuracy to being an Unbalanced Accuracy, depending on its class weight consideration.
- It can range from being a Top-1 Accuracy to being a Top-K Accuracy, depending on its prediction rank consideration.
- It can integrate with Machine Learning Frameworks for automated calculation.
- It can integrate with Model Selection Pipelines for hyperparameter optimization.
- ...
- Example(s):
- Standard Accuracy Metrics, such as:
- A Binary Classification Accuracy of 85.5% for predicting customer response to a promotional campaign.
- A Multi-Class Classification Accuracy of 92% for image classification on CIFAR-10 dataset.
- A Top-5 Accuracy of 95% for ImageNet classification task.
- Balanced Accuracy Variants, such as:
- Balanced Accuracy: Average of True Positive Rate and True Negative Rate.
- Weighted Accuracy: Accuracy weighted by class importance.
- Normalized Accuracy: Accuracy adjusted for chance agreement.
- Domain-Specific Accuracys, such as:
- Temporal Accuracy Metrics, such as:
- Training Accuracy: Accuracy on training data.
- Validation Accuracy: Accuracy on validation data.
- Test Accuracy: Accuracy on held-out test data.
- Ensemble Accuracy Metrics, such as:
- ...
- Standard Accuracy Metrics, such as:
- Counter-Example(s):
- Regression Accuracy Measure, which evaluates continuous predictions rather than categories.
- True Positive Rate, which measures performance on positive class only.
- False Positive Rate, which measures errors on negative class only.
- Cross-Entropy Loss, which measures probabilistic prediction quality.
- Mean Squared Error, which measures continuous prediction errors.
- Ranking Accuracy Measure, which evaluates ordering rather than classification.
- See: Confusion Matrix, Classification Error Rate, Precision, Recall, F-Measure, ROC Curve, AUC, Balanced Accuracy, Cohen's Kappa, Matthews Correlation Coefficient, Cross-Validation, Bootstrap Validation, Statistical Significance, Overfitting, Class Imbalance Problem.
References
2018
- (ML Glossary, 2018) ⇒ (2008). Accuracy. In: Machine Learning Glossary https://developers.google.com/machine-learning/glossary/ Retrieved 2018-04-22.
- QUOTE: The fraction of predictions that a classification model got right. In multi-class classification, accuracy is defined as follows:
[math]\displaystyle{ \text{Accuracy} =\frac{\text{Correct Predictions}} {\text{Total Number Of Examples}} }[/math]
In binary classification, accuracy has the following definition:
[math]\displaystyle{ \text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{\text{Total Number Of Examples}} }[/math]
See true positive and true negative.
- QUOTE: The fraction of predictions that a classification model got right. In multi-class classification, accuracy is defined as follows:
2017
- (Sammut & Webb, 2017) ⇒ (2017) Accuracy. In: Sammut, C., Webb, G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA
- QUOTE: Accuracy refers to a measure of the degree to which the predictions of a model matches the reality being modeled. The term accuracy is often applied in the context of classification models. In this context, [math]\displaystyle{ accuracy = P(\lambda(X) = Y ) }[/math], where [math]\displaystyle{ XY }[/math] is a joint distribution and the classification model [math]\displaystyle{ \lambda }[/math] is a function [math]\displaystyle{ X \rightarrow Y }[/math]. Sometimes, this quantity is expressed as a percentage rather than a value between 0.0 and 1.0.
The accuracy of a model is often assessed or estimated by applying it to test data for which the labels ([math]\displaystyle{ Y }[/math] values) are known. The accuracy of a classifier on test data may be calculated as number of correctly classified objects/total number of objects. Alternatively, a smoothing function may be applied, such as a Laplace estimate or an m-estimate.
Accuracy is directly related to error rate, such that [math]\displaystyle{ accuracy = 1. 0 – error\; rate }[/math] (or when expressed as a percentage, [math]\displaystyle{ accuracy = 100 – error\; rate }[/math]).
- QUOTE: Accuracy refers to a measure of the degree to which the predictions of a model matches the reality being modeled. The term accuracy is often applied in the context of classification models. In this context, [math]\displaystyle{ accuracy = P(\lambda(X) = Y ) }[/math], where [math]\displaystyle{ XY }[/math] is a joint distribution and the classification model [math]\displaystyle{ \lambda }[/math] is a function [math]\displaystyle{ X \rightarrow Y }[/math]. Sometimes, this quantity is expressed as a percentage rather than a value between 0.0 and 1.0.
2002
- (Melli, 2002) ⇒ Gabor Melli. (2002). “PredictionWorks' Data Mining Glossary.
- Accuracy: The measure of a model's ability to correctly label a previously unseen test case. If the label is categorical (classification), accuracy is commonly reported as the rate which a case will be labeled with the right category. For example, a model may be said to predict whether a customer responds to a promotional campaign with 85.5% accuracy. If the label is continuous, accuracy is commonly reported as the average distance between the predicted label and the correct value. For example, a model may be said to predict the amount a customer will spend on a given month within $55. See also Accuracy Estimation, Classification, Estimation, Model, and Statistical Significance.
1998
- (Kohavi & Provost, 1998) ⇒ Ron Kohavi, and Foster Provost. (1998). “Glossary of Terms.” In: Machine Leanring 30(2-3).
- Accuracy (error rate): The rate of correct (incorrect) predictions made by the model over a data set (cf. coverage). Accuracy is usually estimated by using an independent test set that was not used at any time during the learning process. More complex accuracy estimation techniques, such as cross-validation and the bootstrap, are commonly used, especially with data sets containing a small number of instances.