Pairwise Preference Elicitation

A Pairwise Preference Elicitation is a human judgment collection process that is a preference elicitation method for gathering binary preferences between two comparison options.

• Context:
  • It can collect Pairwise Preference Votes through anonymous mechanisms.
  • It can handle Pairwise Preference Ties as valid outcomes.
  • It can mitigate Pairwise Preference Biases through randomization.
  • It can support Pairwise Preference Sampling with non-uniform probabilities.
  • ...
  • It can range from being a Small-Scale Pairwise Preference Elicitation to being a Large-Scale Pairwise Preference Elicitation, depending on its pairwise preference data volume.
  • It can range from being a Synchronous Pairwise Preference Elicitation to being an Asynchronous Pairwise Preference Elicitation, depending on its pairwise preference collection timing.
  • ...
  • It can feed into Pairwise Preference Models like Bradley-Terry Model.
  • It can produce Pairwise Preference Datasets for analysis.
  • ...
• Example(s):
  • LMArena Pairwise Preference Elicitation, comparing LLM responses.
  • Sports Tournament Pairwise Preference Elicitation, based on match outcomes.
  • Product Comparison Pairwise Preference Elicitation, for consumer research.
  • ...
• Counter-Example(s):
  • Multi-Item Ranking Elicitation, which compares more than two items.
  • Absolute Rating Methods, which score items independently.
  • Likert Scale Surveys, which use fixed rating scales.
• See: Bradley-Terry Model, Human Preference Vote, Crowdsourced Evaluation in AI.

---