2015 ExtremeStatesDistributionDecomp

• (Nikolaev et al., 2015) ⇒ Kirill Nikolaev, Alexey Drutsa, Ekaterina Gladkikh, Alexander Ulianov, Gleb Gusev, and Pavel Serdyukov. (2015). “Extreme States Distribution Decomposition Method for Search Engine Online Evaluation.” In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2015). ISBN:978-1-4503-3664-2 doi:10.1145/2783258.2783415

Subject Headings:

Cited By

• http://scholar.google.com/scholar?q=%222015%22+Extreme+States+Distribution+Decomposition+Method+for+Search+Engine+Online+Evaluation
• http://dl.acm.org/citation.cfm?id=2783258.2783415&preflayout=flat#citedby

Quotes

Author Keywords

• A/b test; distribution decomposition; effect variable; evaluation/methodology; human information processing

Abstract

Nowadays, the development of most leading web services is controlled by online experiments that qualify and quantify the steady stream of their updates. The challenging problem is to define an appropriate online metric of user behavior, so-called Overall Evaluation Criterion (OEC), which is both interpretable and sensitive. The state-of-the-art approach is to choose a type of entities to observe in the behavior data, to define a key metric for these observations, and to estimate the average value of this metric over the observations in each of the system versions. A significant disadvantage of the OEC obtained in this way is that the average value of the key metric does not necessarily change, even if its distribution changes significantly. The reason is that the difference between the mean values of the key metric over the two variants of the system does not necessarily reflect the character of the change in the distribution.

We develop a novel method of quantifying the change in the distribution of the key metric, which is (1) interpretable, (2) is based on the analysis of the two distributions as a whole, and, for this reason, is sensitive to more ways the two distributions may actually differ. We provide a thorough theoretical analysis of our approach and show experimentally that, other things being equal, it produces more sensitive OEC than the average.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2015 ExtremeStatesDistributionDecomp	Kirill Nikolaev Alexey Drutsa Ekaterina Gladkikh Alexander Ulianov Gleb Gusev Pavel Serdyukov			Extreme States Distribution Decomposition Method for Search Engine Online Evaluation				10.1145/2783258.2783415		2015