2019 TopKOffPolicyCorrectionforaREIN

• (Chen, Beutel et al., 2019) ⇒ Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. (2019). “Top-k Off-policy Correction for a REINFORCE Recommender System.” In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining.

Subject Headings: YouTube Recommender System.

Cited By

• http://scholar.google.com/scholar?q=%222019%22+Top-k+Off-policy+Correction+for+a+REINFORCE+Recommender+System
• https://dl.acm.org/doi/abs/10.1145/3289600.3290999#sec-ref

Quotes

Abstract

Industrial recommender systems deal with extremely large action spaces -- many millions of items to recommend. Moreover, they need to serve billions of users, who are unique at any point in time, making a complex user state space. Luckily, huge quantities of logged implicit feedback (e.g., user clicks, dwell time) are available for learning. Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the previous versions of the recommender. In this work, we present a general recipe of addressing such biases in a production top-K recommender system at Youtube, built with a policy-gradient-based algorithm, i.e. REINFORCE. The contributions of the paper are: (1) scaling REINFORCE to a production recommender system with an action space on the orders of millions; (2) applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies; (3) proposing a novel top-K off-policy correction to account for our policy recommending multiple items at a time; (4) showcasing the value of exploration. We demonstrate the efficacy of our approaches through a series of simulations and multiple live experiments on Youtube.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2019 TopKOffPolicyCorrectionforaREIN	Minmin Chen Alex Beutel Paul Covington Sagar Jain Francois Belletti Ed H Chi			Top-k Off-policy Correction for a REINFORCE Recommender System