Jump to ContentJump to Main Navigation
Show Summary Details
More options …


Journal of Interactive Media

Editor-in-Chief: Ziegler, Jürgen

3 Issues per year

See all formats and pricing
More options …
Volume 14, Issue 1


Item Familiarity as a Possible Confounding Factor in User-Centric Recommender Systems Evaluation

Dietmar Jannach / Lukas Lerche / Michael Jugovac
Published Online: 2015-04-01 | DOI: https://doi.org/10.1515/icom-2015-0018


User studies play an important role in academic research in the field of recommender systems as they allow us to assess quality factors other than the predictive accuracy of the underlying algorithms. User satisfaction is one such factor that is often evaluated in laboratory settings and in many experimental designs one task of the participants is to assess the suitability of the system-generated recommendations. The effort required by the user to make such an assessment can, however, depend on the user’s familiarity with the presented items and directly impact on the reported user satisfaction. In this paper, we report the results of a preliminary recommender systems user study using Mechanical Turk, which indicates that item familiarity is strongly correlated with overall satisfaction.

Keywords: Recommender Systems; User Study; Satisfaction; Methodology


  • Jannach, D., Zanker, M., Felfernig, A., Friedrich, G.: Recommender Systems - An introduction, Cambridge University Press, 2010.Google Scholar

  • Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems, Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys ‘11), pp. 157–164, 2011Google Scholar

  • Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems, User Modeling and User-Adapted Interaction, 22(4–5), pp. 441–504, 2012CrossrefWeb of ScienceGoogle Scholar

  • Chen L., and Pu, P.: Interaction design guidelines on critiquing-based recommender systems, User Modeling and User-Adapted Interaction 19(3), pp. 167–206, 2009CrossrefWeb of ScienceGoogle Scholar

  • Gedikli, F., Jannach, D., Ge, M.: How should I explain? A comparison of different explanation types for recommender systems, International Journal of Human Computer Studies, 72(4), pp. 367–382, 2014CrossrefWeb of ScienceGoogle Scholar

  • Ekstrand, M. D., Maxwell Harper, F., Willemsen, M. C., Konstan, J. A.: User perception of differences in recommender algorithms, Proceedings of the Eight ACM Conference on Recommender Systems (RecSys ‘14), pp. 161–168, 2014Google Scholar

  • Said, A., Fields, B., Jain, B. J., Albayrak, S.: User-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm, Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ‘14), pp. 1399–1408, 2014Google Scholar

  • Cremonesi, P., Garzotto, F., Turrin, R.: User-centric vs. system-centric evaluation of recommender systems, Proceedings INTERACT 2013, pp. 334–351, 2013.Google Scholar

  • Jannach, D., Lerche, L., Gedikli, G., Bonnin, G.: What recommenders recommend - An analysis of accuracy, popularity, and sales diversity effects, 21st International Conference on User Modeling, Adaptation and Personalization (UMAP ‘13), pp. 25–37, 2013Google Scholar

  • Bettman, J. R., Johnson, E. J., Payne, J. W.: A componential analysis of cognitive effort in choice, Organizational Behavior and Human Decision Processes, 45(1), 111–139, 1990CrossrefGoogle Scholar

  • Payne, J. W., Bettman, J. R., Johnson, E. J.: The Adaptive Decision Maker, Cambridge University Press, 1993Google Scholar

  • McCarthy, K., McGinty, L., Smyth, B., Reilly J.: On the evaluation of dynamic critiquing: A large-scale user study, Twentieth National Conference on Artificial Intelligence (AAAI ‘05), pp. 535–540, 2005Google Scholar

  • Jannach, D., Hegelich, K.: A Case Study on the effectiveness of recommendations in the mobile internet, Proceedings of the Third ACM Conference on Recommender Systems (RecSys ‘09), pp. 205–208, 2009Google Scholar

About the article

Dietmar Jannach

Dietmar Jannach is a Professor in Computer Science at TU Dortmund, Germany, and head of the e-Services Research Group. His main research areas are Recommender Systems and Intelligent Systems in Business.

Lukas Lerche

Lukas Lerche graduated in Computer Science at TU Dortmund, Germany, and is a research associate in the e-Services Research Group. In his Ph.D. studies he focuses on Recommender Systems in e-commerce and explanations in recommendations.

Michael Jugovac

Michael Jugovac graduated in Computer Science at TU Dortmund, Germany, and is a research associate in the e-Services Research Group. In his Ph.D. studies he focuses on Recommender Systems.

Published Online: 2015-04-01

Published in Print: 2015-04-15

Citation Information: icom, Volume 14, Issue 1, Pages 29–39, ISSN (Online) 2196-6826, ISSN (Print) 1618-162X, DOI: https://doi.org/10.1515/icom-2015-0018.

Export Citation

© 2015 Walter de Gruyter GmbH, Berlin/Boston.Get Permission

Comments (0)

Please log in or register to comment.
Log in