2013 PredictiveModelPerformanceOffli

• (Yi et al., 2013) ⇒ Jeonghee Yi, Ye Chen, Jie Li, Swaraj Sett, and Tak W. Yan. (2013). “Predictive Model Performance: Offline and Online Evaluations.” In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ISBN:978-1-4503-2174-7 doi:10.1145/2487575.2488215

Subject Headings:

Cited By

• http://scholar.google.com/scholar?q=%222013%22+Predictive+Model+Performance%3A+Offline+and+Online+Evaluations
• http://dl.acm.org/citation.cfm?id=2487575.2488215&preflayout=flat#citedby

Quotes

Author Keywords

• Auc; click prediction; log-likelihood; miscellaneous; model evaluation metric; offline evaluation; online advertising; online evaluation; performance measures; prediction error; rig; simulated metric; sponsored search

Abstract

We study the accuracy of evaluation metrics used to estimate the efficacy of predictive models. Offline evaluation metrics are indicators of the expected model performance on real data. However, in practice we often experience substantial discrepancy between the offline and online performance of the models.

We investigate the characteristics and behaviors of the evaluation metrics on offline and online testing both analytically and empirically by experimenting them on online advertising data from the Bing search engine. One of our findings is that some offline metrics like AUC (the Area Under the Receiver Operating Characteristic Curve) and RIG (Relative Information Gain) that summarize the model performance on the entire spectrum of operating points could be quite misleading sometimes and result in significant discrepancy in offline and online metrics. For example, for click prediction models for search advertising, errors in predictions in the very low range of predicted click scores impact the online performance much more negatively than errors in other regions. Most of the offline metrics we studied including AUC and RIG, however, are insensitive to such model behavior.

We designed a new model evaluation paradigm that simulates the online behavior of predictive models. For a set of ads selected by a new prediction model, the online user behavior is estimated from the historic user behavior in the search logs. The experimental results on click prediction model for search advertising are highly promising.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2013 PredictiveModelPerformanceOffli	Ye Chen Tak W. Yan Jeonghee Yi Jie Li Swaraj Sett			Predictive Model Performance: Offline and Online Evaluations				10.1145/2487575.2488215		2013