2010 OverlappingExperimentInfrastruc

• (Tang et al., 2010) ⇒ Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. (2010). “Overlapping Experiment Infrastructure: More, Better, Faster Experimentation.” In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2010). doi:10.1145/1835804.1835810

Subject Headings: AB Testing.

Notes

• Video Lecture: http://videolectures.net/kdd2010_tang_oei/
• Categories and Subject Descriptors: G.3 Probability and Statistics: Experimental Design — controlled experiments, randomized experiments, A/B testing; I.2.6 Learning: real-time, automation, causality.
• General Terms: Measurement, Design, Experimentation, Human Factors, Performance

Cited By

• http://scholar.google.com/scholar?q=%22Overlapping+experiment+infrastructure%3A+more%2C+better%2C+faster+experimentation%22+2010
• http://portal.acm.org/citation.cfm?id=1835810&preflayout=flat#citedby

Quotes

Author Keywords

Controlled experiments, A/B testing, Website Testing, MultiVariable Testing

Abstract

At Google, experimentation is practically a mantra; we evaluate almost every change that potentially affects what our users experience. Such changes include not only obvious user-visible changes such as modifications to a user interface, but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection. Our insatiable appetite for experimentation has led us to tackle the problems of how to run more experiments, how to run experiments that produce better decisions, and how to run them faster. In this paper, we describe Google's overlapping experiment infrastructure that is a key component to solving these problems. In addition, because an experiment infrastructure alone is insufficient, we also discuss the associated tools and educational processes required to use it effectively. We conclude by describing trends that show the success of this overall experimental environment. While the paper specifically describes the experiment system and experimental processes we have in place at Google, we believe they can be generalized and applied by any entity interested in using experimentation to improve search engines and other web applications.

References

,

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2010 OverlappingExperimentInfrastruc	Diane Tang Ashish Agarwal Deirdre O'Brien Mike Meyer			Overlapping Experiment Infrastructure: More, Better, Faster Experimentation		KDD-2010 Proceedings	http://static.googleusercontent.com/external content/untrusted dlcp/research.google.com/en//pubs/archive/36500.pdf	10.1145/1835804.1835810		2010