Crowd Adjudication Process

A Crowd Adjudication Process is an evaluation process that resolves evaluation disagreements through crowd workers applying majority voting or aggregation rules to achieve consensus decisions.

• AKA: Crowdsourced Adjudication Process, Crowd Resolution Process, Non-Expert Adjudication, Crowd-Based Disagreement Resolution.
• Context:
  • It can typically employ Multiple Crowd Workers for statistical robustness.
  • It can typically use Majority Vote Rules for conflict resolution.
  • It can often incorporate Quality Control Checks filtering unreliable workers.
  • It can often apply Weighted Aggregation based on worker reputation.
  • It can reduce Adjudication Costs compared to expert adjudication.
  • It can enable Scalable Resolution for large-scale annotation.
  • It can integrate with Crowdsourcing Platforms like Amazon MTurk.
  • It can generate Worker Agreement Statistics for quality monitoring.
  • It can range from being a Simple Majority Adjudication to being a Weighted Consensus Adjudication, depending on its aggregation complexity.
  • It can range from being a Binary Crowd Adjudication to being a Multi-Class Crowd Adjudication, depending on its label type.
  • It can range from being a Single-Round Adjudication to being a Iterative Adjudication, depending on its resolution depth.
  • It can range from being a Open Crowd Adjudication to being a Qualified Crowd Adjudication, depending on its worker requirements.
  • ...
• Examples:
  • Platform-Based Adjudications, such as:
    • MTurk Adjudication Process for scalable resolution.
    • Figure Eight Adjudication for quality control.
  • Task-Specific Adjudications, such as:
    • Image Label Adjudication for vision tasks.
    • Text Classification Adjudication for NLP tasks.
  • Hybrid Adjudications, such as:
    • Crowd-Expert Hybrid Adjudication combining crowd scale with expert validation.
  • ...
• Counter-Examples:
  • Expert Adjudication Process, which uses domain experts.
  • Automated Adjudication Process, which lacks human judgment.
  • Individual Resolution, which avoids crowd consensus.
• See: Evaluation Process, Expert Adjudication Process, Crowd Agreement Metric, Crowdsourcing Platform, Majority Voting, Worker Quality Control, Evaluation Resolution Process.

---