Rationale-Based Classification Measure

A Rationale-Based Classification Measure is a classification performance measure that is an interpretability metric evaluating both classification accuracy and rationale quality in rationale-guided text classification systems.

• AKA: Rationale-Aware Performance Metric, Select-Then-Classify Evaluation Measure.
• Context:
  • It can typically assess Rationale Plausibility via human agreement scores.
  • It can typically measure Rationale Faithfulness through comprehensiveness and sufficiency.
  • It can typically evaluate Rationale Conciseness using length penaltys.
  • It can typically quantify Classification Performance conditioned on rationale quality.
  • It can typically detect Degenerate Rationales that include entire input.
  • ...
  • It can often combine Multiple Evaluation Dimensions in composite scores.
  • It can often incorporate Adversarial Tests for faithfulness verification.
  • It can often use Gradient-Based Methods for importance comparison.
  • It can often employ Human Evaluations for plausibility assessment.
  • ...
  • It can range from being an Automatic Rationale Measure to being a Human-Evaluated Rationale Measure, depending on its evaluation method.
  • It can range from being a Binary Rationale Measure to being a Continuous Rationale Measure, depending on its scoring granularity.
  • ...
  • It can evaluate Rationale-Guided Text Classification Task performance.
  • It can diagnose Rationale Quality Issues in classification systems.
  • It can guide Model Selection for interpretable NLP applications.
  • It can support Interpretability Research in machine learning.
  • ...
• Example(s):
  • Comprehensiveness Score, measuring prediction drop when rationale removed.
  • Sufficiency Score, measuring prediction confidence with only rationale.
  • Rationale F1 Score, comparing with human-annotated rationales.
  • AUPRC Score, evaluating rationale ranking quality.
  • Plausibility-Faithfulness Score, combining human agreement and model behavior.
  • ...
• Counter-Example(s):
  • Pure Accuracy Metrics, which ignore rationale quality.
  • Attention Weight Analysis, which examines soft attention not discrete rationales.
  • Post-hoc Explanation Metrics, which evaluate generated explanations not selected rationales.
• See: Interpretability Metric, Explainability Evaluation, Faithfulness Measure, Classification Performance Metric.

---