Item Recommendations System Evaluation Task

A Item Recommendations System Evaluation Task is an information system evaluation task for an item recommendation system.

• Context:
  • …
• Example(s):
  • One based on an MovieLens Benchmark.
  • …
• Counter-Example(s):
  • NLP Evaluation Task.
• See: IR System Evaluation, Item Recommendation Performance Measure.

---

---

References

2019

• (Rendle et al., 2019) ⇒ Steffen Rendle, Li Zhang, and Yehuda Koren. (2019). “On the Difficulty of Evaluating Baselines: A Study on Recommender Systems.” In: arXiv preprint arXiv:1905.01395.
  • ABSTRACT: ... Numerical evaluations with comparisons to baselines play a central role when judging research in recommender systems. In this paper, we show that running baselines properly is difficult. We demonstrate this issue on two extensively studied datasets. First, we show that results for baselines that have been used in numerous publications over the past five years for the Movielens 10M benchmark are suboptimal. With a careful setup of a vanilla matrix factorization baseline, we are not only able to improve upon the reported results for this baseline but even outperform the reported results of any newly proposed method. Secondly, we recap the tremendous effort that was required by the community to obtain high quality results for simple methods on the Netflix Prize. . Our results indicate that empirical findings in research papers are questionable unless they were obtained on standardized benchmarks where baselines have been tuned extensively by the research community.

---