Skip to content
Licensed Unlicensed Requires Authentication Published by Oldenbourg Wissenschaftsverlag April 1, 2015

Item Familiarity as a Possible Confounding Factor in User-Centric Recommender Systems Evaluation

Dietmar Jannach, Lukas Lerche and Michael Jugovac
From the journal i-com


User studies play an important role in academic research in the field of recommender systems as they allow us to assess quality factors other than the predictive accuracy of the underlying algorithms. User satisfaction is one such factor that is often evaluated in laboratory settings and in many experimental designs one task of the participants is to assess the suitability of the system-generated recommendations. The effort required by the user to make such an assessment can, however, depend on the user’s familiarity with the presented items and directly impact on the reported user satisfaction. In this paper, we report the results of a preliminary recommender systems user study using Mechanical Turk, which indicates that item familiarity is strongly correlated with overall satisfaction.


Jannach, D., Zanker, M., Felfernig, A., Friedrich, G.: Recommender Systems - An introduction, Cambridge University Press, 2010.10.1017/CBO9780511763113Search in Google Scholar

Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems, Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys ‘11), pp. 157–164, 201110.1145/2043932.2043962Search in Google Scholar

Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems, User Modeling and User-Adapted Interaction, 22(4–5), pp. 441–504, 201210.1007/s11257-011-9118-4Search in Google Scholar

Chen L., and Pu, P.: Interaction design guidelines on critiquing-based recommender systems, User Modeling and User-Adapted Interaction 19(3), pp. 167–206, 200910.1007/s11257-008-9057-xSearch in Google Scholar

Gedikli, F., Jannach, D., Ge, M.: How should I explain? A comparison of different explanation types for recommender systems, International Journal of Human Computer Studies, 72(4), pp. 367–382, 201410.1016/j.ijhcs.2013.12.007Search in Google Scholar

Ekstrand, M. D., Maxwell Harper, F., Willemsen, M. C., Konstan, J. A.: User perception of differences in recommender algorithms, Proceedings of the Eight ACM Conference on Recommender Systems (RecSys ‘14), pp. 161–168, 201410.1145/2645710.2645737Search in Google Scholar

Said, A., Fields, B., Jain, B. J., Albayrak, S.: User-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm, Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ‘14), pp. 1399–1408, 201410.1145/2441776.2441933Search in Google Scholar

Cremonesi, P., Garzotto, F., Turrin, R.: User-centric vs. system-centric evaluation of recommender systems, Proceedings INTERACT 2013, pp. 334–351, 2013.10.1007/978-3-642-40477-1_21Search in Google Scholar

Jannach, D., Lerche, L., Gedikli, G., Bonnin, G.: What recommenders recommend - An analysis of accuracy, popularity, and sales diversity effects, 21st International Conference on User Modeling, Adaptation and Personalization (UMAP ‘13), pp. 25–37, 201310.1007/978-3-642-38844-6_3Search in Google Scholar

Bettman, J. R., Johnson, E. J., Payne, J. W.: A componential analysis of cognitive effort in choice, Organizational Behavior and Human Decision Processes, 45(1), 111–139, 199010.1016/0749-5978(90)90007-VSearch in Google Scholar

Payne, J. W., Bettman, J. R., Johnson, E. J.: The Adaptive Decision Maker, Cambridge University Press, 199310.1017/CBO9781139173933Search in Google Scholar

McCarthy, K., McGinty, L., Smyth, B., Reilly J.: On the evaluation of dynamic critiquing: A large-scale user study, Twentieth National Conference on Artificial Intelligence (AAAI ‘05), pp. 535–540, 2005Search in Google Scholar

Jannach, D., Hegelich, K.: A Case Study on the effectiveness of recommendations in the mobile internet, Proceedings of the Third ACM Conference on Recommender Systems (RecSys ‘09), pp. 205–208, 200910.1145/1639714.1639749Search in Google Scholar

Published Online: 2015-04-01
Published in Print: 2015-04-15

© 2015 Walter de Gruyter GmbH, Berlin/Boston