Selective Classification Performance Measure

A Selective Classification Performance Measure is a classification performance metric that is an efficiency-aware metric evaluating both classification accuracy and selection quality in selective text classification systems.

• AKA: Sparse Classification Metric, Selection-Aware Performance Measure.
• Context:
  • It can typically assess Selection Sparsity versus classification accuracy.
  • It can typically measure Relevance Precision of selected spans.
  • It can typically quantify Information Retention in sparse selection.
  • It can typically evaluate Selection Consistency across similar inputs.
  • It can typically penalize Over-Selection degrading interpretability.
  • ...
  • It can often incorporate Trade-off Curves between sparsity and accuracy.
  • It can often employ Human Evaluation for selection quality.
  • It can often measure Computational Savings from selective processing.
  • It can often include Robustness Tests on selection stability.
  • ...
  • It can range from being a Joint Selection-Classification Measure to being a Decomposed Component Measure, depending on its evaluation approach.
  • It can range from being a Threshold-Based Measure to being a Continuous Measure, depending on its scoring method.
  • ...
  • It can evaluate Selective Text Classification Task performance.
  • It can guide Sparsity-Accuracy Trade-offs in model design.
  • It can diagnose Selection Problems versus classification errors.
  • It can support Efficiency Optimization in production systems.
  • ...
• Example(s):
  • Sparsity-Adjusted Accuracy, normalizing by selection percentage.
  • Selection F1 Score, comparing with human-selected spans.
  • Information-Theoretic Measure, quantifying mutual information.
  • Pareto Efficiency Score, balancing multiple objectives.
  • Stability-Aware Metric, measuring selection variance.
  • ...
• Counter-Example(s):
  • Pure Accuracy Metrics, which ignore selection constraint.
  • Dense Attention Scores, which measure full attention weights.
  • Throughput Metrics, which measure speed not selection quality.
• See: Efficiency Metric, Interpretability Measure, Sparse Processing Evaluation, Multi-Objective Metric.

---