Prediction Task Performance Measure

A Prediction Task Performance Measure is a performance measure for prediction tasks that quantifies prediction error rates and prediction accuracy.

• AKA: Predictive Performance Measure, Prediction Error Measure, Predictive Error Measure, Error Rate Measure, Error Rate Metric, Prediction Accuracy Measure.
• Context:
  • It can (typically) quantify Type I Errors (false positives) and Type II Errors (false negatives) in classification tasks.
  • It can (typically) be calculated by a Predictive Performance Measuring System using test data or validation data.
  • It can (typically) incorporate cost-sensitive metrics when misclassification costs are asymmetric.
  • It can (often) be used in Model Selection Tasks to compare predictive models.
  • It can (often) be optimized during Hyperparameter Tuning Tasks.
  • It can (often) be computed using Cross-Validation or Holdout Validation methods.
  • It can (often) require Baseline Performance Measures for meaningful interpretation.
  • It can range from being a Classification Performance Measure to being a Ranking Performance Measure to being a Numeric Prediction Performance Measure, depending on its prediction task type.
  • It can range from being a Binary Prediction Performance Measure to being a Multiclass Prediction Performance Measure, depending on its class count.
  • It can range from being a Point Estimate Performance Measure to being a Probabilistic Performance Measure, depending on its prediction type.
  • It can range from being a Threshold-Dependent Performance Measure to being a Threshold-Independent Performance Measure, depending on its decision threshold sensitivity.
  • It can integrate with Machine Learning Platforms for automated evaluation.
  • It can integrate with A/B Testing Systems for model comparison.
  • ...
• Example(s):
  • Classification Performance Measures, such as:
    • Accuracy Measure: Proportion of correct predictions.
    • Precision Measure: Proportion of positive predictions that are correct.
    • Recall Measure: Proportion of actual positives correctly identified.
    • F1 Score: Harmonic mean of precision and recall.
    • Matthews Correlation Coefficient: Correlation between predicted and actual classes.
    • Cohen's Kappa: Agreement corrected for chance.
    • Area Under ROC Curve (AUC-ROC): Threshold-independent classification performance.
    • Area Under Precision-Recall Curve (AUC-PR): Performance on imbalanced datasets.
    • Log Loss: Probabilistic prediction accuracy.
    • Brier Score: Mean squared difference between predicted probabilities and outcomes.
  • Ranking Performance Measures, such as:
    • Normalized Discounted Cumulative Gain (NDCG): Quality of ranking with graded relevance.
    • Mean Average Precision (MAP): Average precision across queries.
    • Mean Reciprocal Rank (MRR): Average reciprocal of first relevant result rank.
    • Precision at K: Precision in top K results.
    • Kendall's Tau: Correlation between predicted and actual rankings.
  • Regression Performance Measures, such as:
    • Mean Absolute Error (MAE): Average absolute prediction error.
    • Mean Squared Error (MSE): Average squared prediction error.
    • Root Mean Squared Error (RMSE): Square root of MSE.
    • Mean Absolute Percentage Error (MAPE): Percentage-based error metric.
    • R-Squared (R²): Proportion of variance explained.
    • Adjusted R-Squared: R² adjusted for number of predictors.
    • Median Absolute Error: Robust to outliers.
  • Clustering Performance Measures, such as:
    • Silhouette Score: Cluster cohesion and separation.
    • Davies-Bouldin Index: Ratio of within-cluster to between-cluster distance.
    • Calinski-Harabasz Index: Ratio of between-group to within-group dispersion.
  • Sequence Prediction Performance Measures, such as:
    • Word Error Rate (WER): Error rate in speech recognition.
    • BLEU Score: Machine translation quality.
    • ROUGE Score: Text summarization quality.
    • Character Error Rate (CER): Character-level prediction accuracy.
  • Time Series Performance Measures, such as:
    • Mean Absolute Scaled Error (MASE): Scale-independent forecast accuracy.
    • Symmetric Mean Absolute Percentage Error (SMAPE): Symmetric percentage error.
    • Theil's U Statistic: Relative forecast accuracy.
  • Domain-Specific Performance Measures, such as:
    • Intersection over Union (IoU): Object detection accuracy.
    • Dice Coefficient: Image segmentation overlap.
    • Perplexity: Language model quality.
    • Click-Through Rate (CTR): Recommendation system performance.
  • ...
• Counter-Example(s):
  • Success Rate Metric, which measures positive outcomes rather than errors.
  • Computational Performance Measure, which measures efficiency rather than accuracy.
  • Data Quality Measure, which assesses input data rather than predictions.
  • Model Complexity Measure, which quantifies model structure rather than performance.
  • Training Convergence Measure, which tracks optimization progress rather than final accuracy.
• See: Predictive Model, Model Evaluation Task, Confusion Matrix, ROC Curve, Precision-Recall Curve, Cross-Validation, Overfitting, Underfitting, Bias-Variance Tradeoff, Statistical Hypothesis Testing Task, Type I Error, Type II Error, Predictive Feature, Feature Importance, Model Selection, Hyperparameter Optimization.

---

---

References

2017

• https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html
  • QUOTE: spark.mllib comes with a number of machine learning algorithms that can be used to learn from and make predictions on data. When these algorithms are applied to build machine learning models, there is a need to evaluate the performance of the model on some criteria, which depends on the application and its requirements. spark.mllib also provides a suite of metrics for the purpose of evaluating the performance of machine learning models.

    Specific machine learning algorithms fall under broader types of machine learning applications like classification, regression, clustering, etc. Each of these types have well established metrics for performance evaluation and those metrics that are currently available in spark.mllib are detailed in this section.

2011

• (Kai Ming Ting, 2011b) ⇒ Kai Ming Ting. (2011). “Error Rate.” In: (Sammut & Webb, 2011) p.331

1998

• (Kohavi & Provost, 1998) ⇒ Ron Kohavi, and Foster Provost. (1998). “Glossary of Terms.” In: Machine Leanring 30(2-3).
  • Error rate: See Accuracy.

1983

• (Efron, 1983) ⇒ Bradley Efron, (1983). “Estimating the error rate of a prediction rule: improvement on cross-validation.” In: Journal of the American Statistical Association, 78(382). http://www.jstor.org/stable/2288636
  • QUOTE: We construct a prediction rule on the basis of some data, and then wish to estimate the error rate of this rule in classifying future observations. Cross-validation provides a nearly unbiased estimate, using only the original data. Cross-validation turns out to be related closely to the bootstrap estimate of the error rate. This article has two purposes: to understand better the theoretical basis of the prediction problem, and to investigate some related estimators, which seem to offer considerably improved estimation in small samples.

---