Crowd Agreement Measure

A Crowd Agreement Measure is an evaluation reliability measure that quantifies consensus levels among crowd workers accounting for worker quality variation and task difficulty.

AKA: Crowd Consensus Measure, Worker Agreement Measure, Crowd Reliability Measure, Non-Expert Agreement Index.
Context:
- It can typically incorporate Worker Quality Weights in agreement calculation.
- It can typically handle Sparse Annotation Matrixes from partial overlap.
- It can often detect Spammer Workers through agreement patterns.
- It can often adjust for Task Difficulty affecting expected agreement.
- It can support Quality Control Decisions in crowdsourcing workflows.
- It can enable Worker Performance Tracking for payment systems.
- It can facilitate Annotation Aggregation weighting reliable workers.
- It can integrate with Active Learning selecting ambiguous instances.
- It can range from being a Binary Crowd Agreement to being a Multi-Class Crowd Agreement, depending on its label type.
- It can range from being a Raw Crowd Agreement to being a Adjusted Crowd Agreement, depending on its correction method.
- It can range from being a Pairwise Crowd Agreement to being a Group Crowd Agreement, depending on its worker scope.
- It can range from being a Task-Level Agreement to being a Instance-Level Agreement, depending on its granularity.
- ...
Examples:
- Statistical Agreement Measures, such as:
  - Fleiss Kappa for Crowds handling multiple raters.
  - Weighted Majority Agreement using worker reputation.
  - MACE Agreement Score modeling worker competence.
- Platform-Specific Measures, such as:
  - MTurk Agreement Score for Amazon Mechanical Turk.
  - Figure Eight Agreement with quality control.
- Task-Specific Agreements, such as:
  - Image Annotation Agreement for vision tasks.
  - Text Classification Agreement for NLP tasks.
- ...
Counter-Examples:
- Inter-Expert Agreement Measure, which measures expert consensus.
- Automated Agreement Score, which lacks human annotation.
- Individual Worker Measure, which ignores collective agreement.
See: Evaluation Reliability Measure, Inter-Expert Agreement Measure, Crowdsourcing Quality Control, Worker Quality Measure, Crowd Adjudication Process, Agreement Coefficient, Crowd Annotation.