Text Annotator Bias Measure

A Text Annotator Bias Measure is a Task Error Measure that measures the similarity of a Text Annotator Performance to a desired Task Performance Measure.

• AKA: Annotator Bias.
• Context:
  • It can (typically) be reported for human annotators.
• See: Annotator Bias Correction, Text Annotation Task, Annotation Agreement.

---

---

References

2008

• (Snow et al., 2008) ⇒ Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. (2008). “Cheap and Fast - But is it Good?: Evaluating non-expert annotations for natural language tasks.” In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2008).
  • QUOTE: A wide number of methods have been explored to correct for the bias of annotators. Dawid and Skene (1979) are the first to consider the case of having multiple annotators per example but unknown true labels. They introduce an EM algorithm to simultaneously estimate annotator biases and latent label classes. Wiebe et al. (1999) analyze |linguistic annotator agreement statistics to find bias, and use a similar model to correct labels. A large literature in biostatistics addresses this same problem for medical diagnosis. Albert and Dodd (2004) review several related models, but argue they have various shortcomings and emphasize instead the importance of having a gold standard.

2004

• (Albert & Dodd, 2004) ⇒ Paul S. Albert and Lori E. Dodd. (2004). A Cautionary Note on the Robustness of Latent Class Models for Estimating Diagnostic Error without a Gold Standard. Biometrics, Vol. 60 (2004), pp. 427-435.

1999

• (Wiebe et al., 1999) ⇒ Janyce M. Wiebe, Rebecca F. Bruce and Thomas P. O’Hara. (1999). Development and use of a goldstandard data set for subjectivity classifications. In: Proceedings of ACL-1999.

1979

• (Dawid & Skene, 1979) ⇒ A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, Vol. 28, No. 1 (1979), pp. 20-28.

---