2009 EfficientlyLearningtheAccuracyo

(Donmez et al., 2009) ⇒ Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider. (2009). “Efficiently Learning the Accuracy of Labeling Sources for Selective Sampling.” In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2009). doi:10.1145/1557019.1557053

Subject Headings:

Notes

Categories and Subject Descriptors: I.5.2 Pattern Recognition: Design Methodology — Classifier design and evaluation; H.2.8 Database Applications: Data mining.
General Terms: Algorithms, Design, Experimentation, Performance, Measurement

Cited By

Quotes

Author Keywords

Noisy Labelers, Estimation, Active Learning, Labeler Selection

Abstract

Many scalable data mining tasks rely on active learning to provide the most useful accurately labeled instances. However, what if there are multiple labeling sources (`oracles' or `experts') with different but unknown reliabilities? With the recent advent of inexpensive and scalable online annotation tools, such as Amazon's Mechanical Turk, the labeling process has become more vulnerable to noise - and without prior knowledge of the accuracy of each individual labeler. This paper addresses exactly such a challenge : how to jointly learn the accuracy of labeling sources and obtain the most informative labels for the active learning task at hand minimizing total labeling effort. More specifically, we present IEThresh (Interval Estimate Threshold) as a strategy to intelligently select the expert(s) with the highest estimated labeling accuracy. IEThresh estimates a confidence interval for the reliability of each expert and filters out the one(s) whose estimated upper-bound confidence interval is below a threshold - which jointly optimizes expected accuracy (mean) and need to better estimate the expert's accuracy (variance). Our framework is flexible enough to work with a wide range of different noise levels and outperforms baselines such as asking all available experts and random expert selection. In particular, IEThresh achieves a given level of accuracy with less than half the queries issued by all-experts labeling and less than a third the queries required by random expert selection on datasets such as the UCI mushroom one. The results show that our method naturally balances exploration and exploitation as it gains knowledge of which experts to rely upon, and selects them with increasing frequency.

References

,

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2009 EfficientlyLearningtheAccuracyo	Pinar Donmez Jaime G. Carbonell (1953 – 2020) Jeff Schneider			Efficiently Learning the Accuracy of Labeling Sources for Selective Sampling		KDD-2009 Proceedings		10.1145/1557019.1557053		2009