2013 EvaluatingtheCrowdwithConfidenc

• (Joglekar et al., 2013) ⇒ Manas Joglekar, Hector Garcia-Molina, and Aditya Parameswaran. (2013). “Evaluating the Crowd with Confidence.” In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ISBN:978-1-4503-2174-7 doi:10.1145/2487575.2487595

Subject Headings:

Cited By

• http://scholar.google.com/scholar?q=%222013%22+Evaluating+the+Crowd+with+Confidence
• http://dl.acm.org/citation.cfm?id=2487575.2487595&preflayout=flat#citedby

Quotes

Author Keywords

• Confidence; crowdsourcing; general

Abstract

Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2013 EvaluatingtheCrowdwithConfidenc	Hector Garcia-Molina Aditya Parameswaran Manas Joglekar			Evaluating the Crowd with Confidence				10.1145/2487575.2487595		2013