Item Familiarity as a Possible Confounding Factor in User-Centric Recommender Systems Evaluation

Dietmar Jannach 1 , Lukas Lerche 1 , and Michael Jugovac 1
  • 1 TU Dortmund, Germany
Dietmar Jannach
  • Corresponding author
  • Email
  • Further information
  • Dietmar Jannach is a Professor in Computer Science at TU Dortmund, Germany, and head of the e-Services Research Group. His main research areas are Recommender Systems and Intelligent Systems in Business.
  • Search for other articles:
  • degruyter.comGoogle Scholar
, Lukas Lerche
  • Email
  • Further information
  • Lukas Lerche graduated in Computer Science at TU Dortmund, Germany, and is a research associate in the e-Services Research Group. In his Ph.D. studies he focuses on Recommender Systems in e-commerce and explanations in recommendations.
  • Search for other articles:
  • degruyter.comGoogle Scholar
and Michael Jugovac
  • Email
  • Further information
  • Michael Jugovac graduated in Computer Science at TU Dortmund, Germany, and is a research associate in the e-Services Research Group. In his Ph.D. studies he focuses on Recommender Systems.
  • Search for other articles:
  • degruyter.comGoogle Scholar

Abstract

User studies play an important role in academic research in the field of recommender systems as they allow us to assess quality factors other than the predictive accuracy of the underlying algorithms. User satisfaction is one such factor that is often evaluated in laboratory settings and in many experimental designs one task of the participants is to assess the suitability of the system-generated recommendations. The effort required by the user to make such an assessment can, however, depend on the user’s familiarity with the presented items and directly impact on the reported user satisfaction. In this paper, we report the results of a preliminary recommender systems user study using Mechanical Turk, which indicates that item familiarity is strongly correlated with overall satisfaction.

  • Jannach, D., Zanker, M., Felfernig, A., Friedrich, G.: Recommender Systems - An introduction, Cambridge University Press, 2010.

    • Crossref
    • Export Citation
  • Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems, Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys ‘11), pp. 157–164, 2011

    • Crossref
    • Export Citation
  • Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems, User Modeling and User-Adapted Interaction, 22(4–5), pp. 441–504, 2012

    • Crossref
    • Export Citation
  • Chen L., and Pu, P.: Interaction design guidelines on critiquing-based recommender systems, User Modeling and User-Adapted Interaction 19(3), pp. 167–206, 2009

    • Crossref
    • Export Citation
  • Gedikli, F., Jannach, D., Ge, M.: How should I explain? A comparison of different explanation types for recommender systems, International Journal of Human Computer Studies, 72(4), pp. 367–382, 2014

    • Crossref
    • Export Citation
  • Ekstrand, M. D., Maxwell Harper, F., Willemsen, M. C., Konstan, J. A.: User perception of differences in recommender algorithms, Proceedings of the Eight ACM Conference on Recommender Systems (RecSys ‘14), pp. 161–168, 2014

    • Crossref
    • Export Citation
  • Said, A., Fields, B., Jain, B. J., Albayrak, S.: User-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm, Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ‘14), pp. 1399–1408, 2014

    • Crossref
    • Export Citation
  • Cremonesi, P., Garzotto, F., Turrin, R.: User-centric vs. system-centric evaluation of recommender systems, Proceedings INTERACT 2013, pp. 334–351, 2013.

    • Crossref
    • Export Citation
  • Jannach, D., Lerche, L., Gedikli, G., Bonnin, G.: What recommenders recommend - An analysis of accuracy, popularity, and sales diversity effects, 21st International Conference on User Modeling, Adaptation and Personalization (UMAP ‘13), pp. 25–37, 2013

    • Crossref
    • Export Citation
  • Bettman, J. R., Johnson, E. J., Payne, J. W.: A componential analysis of cognitive effort in choice, Organizational Behavior and Human Decision Processes, 45(1), 111–139, 1990

    • Crossref
    • Export Citation
  • Payne, J. W., Bettman, J. R., Johnson, E. J.: The Adaptive Decision Maker, Cambridge University Press, 1993

    • Crossref
    • Export Citation
  • McCarthy, K., McGinty, L., Smyth, B., Reilly J.: On the evaluation of dynamic critiquing: A large-scale user study, Twentieth National Conference on Artificial Intelligence (AAAI ‘05), pp. 535–540, 2005

  • Jannach, D., Hegelich, K.: A Case Study on the effectiveness of recommendations in the mobile internet, Proceedings of the Third ACM Conference on Recommender Systems (RecSys ‘09), pp. 205–208, 2009

Purchase article
Get instant unlimited access to the article.
$42.00
Log in
Already have access? Please log in.


or
Log in with your institution

Journal + Issues

Search