2010 OverlappingExperimentInfrastruc
- (Tang et al., 2010) ⇒ Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. (2010). “Overlapping Experiment Infrastructure: More, Better, Faster Experimentation.” In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2010). doi:10.1145/1835804.1835810
Subject Headings: AB Testing.
Notes
- Video Lecture: http://videolectures.net/kdd2010_tang_oei/
- Categories and Subject Descriptors: G.3 Probability and Statistics: Experimental Design — controlled experiments, randomized experiments, A/B testing; I.2.6 Learning: real-time, automation, causality.
- General Terms: Measurement, Design, Experimentation, Human Factors, Performance
Cited By
- http://scholar.google.com/scholar?q=%22Overlapping+experiment+infrastructure%3A+more%2C+better%2C+faster+experimentation%22+2010
- http://portal.acm.org/citation.cfm?id=1835810&preflayout=flat#citedby
Quotes
Author Keywords
Controlled experiments, A/B testing, Website Testing, MultiVariable Testing
Abstract
At Google, experimentation is practically a mantra; we evaluate almost every change that potentially affects what our users experience. Such changes include not only obvious user-visible changes such as modifications to a user interface, but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection. Our insatiable appetite for experimentation has led us to tackle the problems of how to run more experiments, how to run experiments that produce better decisions, and how to run them faster. In this paper, we describe Google's overlapping experiment infrastructure that is a key component to solving these problems. In addition, because an experiment infrastructure alone is insufficient, we also discuss the associated tools and educational processes required to use it effectively. We conclude by describing trends that show the success of this overall experimental environment. While the paper specifically describes the experiment system and experimental processes we have in place at Google, we believe they can be generalized and applied by any entity interested in using experimentation to improve search engines and other web applications.
References
,
Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
---|---|---|---|---|---|---|---|---|---|---|
2010 OverlappingExperimentInfrastruc | Diane Tang Ashish Agarwal Deirdre O'Brien Mike Meyer | Overlapping Experiment Infrastructure: More, Better, Faster Experimentation | KDD-2010 Proceedings | http://static.googleusercontent.com/external content/untrusted dlcp/research.google.com/en//pubs/archive/36500.pdf | 10.1145/1835804.1835810 | 2010 |