Skip to content
Licensed Unlicensed Requires Authentication Published by Oldenbourg Wissenschaftsverlag April 22, 2021

A User-Centered Approach to Gamify the Manual Creation of Training Data for Machine Learning

Sarah Alaghbari, Annett Mitschick ORCID logo, Gregor Blichmann, Martin Voigt and Raimund Dachselt ORCID logo
From the journal i-com


The development of artificial intelligence, e. g. for Computer Vision, through supervised learning requires the input of large amounts of annotated or labeled data objects as training data. Usually, the creation of high-quality training data is done manually which can be repetitive and tiring. Gamification, the use of game elements in a non-game context, is one method to make such tedious tasks more interesting. We propose a multi-step process for gamifying the manual creation of training data for machine learning purposes. In this article, we give an overview of related concepts and existing implementations and present a user-centered approach for a real-life use case. Based on a survey within the target user group we identified annotation use cases and dominant player characteristics. The results served as a foundation for designing the gamification concepts which were then discussed with the participants. The final concept includes levels of increasing difficulty, tutorials, progress indicators and a narrative built around a robot character which at the same time is a user assistant. The implemented prototype is an extension of an existing annotation tool at an AI product company and serves as a basis for further observations.

Funding source: Bundesministerium für Bildung und Forschung

Award Identifier / Grant number: 16SV8396

Funding statement: This work was partially funded by the German Research Foundation (DFG) as part of Germany’s Excellence Strategy – EXC 2050/1 – Project ID 390696704 – Cluster of Excellence “Centre for Tactile Internet with Human-in-the-Loop” (CeTI) of Technische Universität Dresden, and by DFG grant 389792660 as part of TRR 248 (see Additionally, the work at AI4BD was funded by the Federal Ministry of Education and Research (BMBF) and the research project PANDIA (grant number 16SV8396).


[1] Maximilian Altmeyer, Pascal Lessel, and Antonio Krüger. 2016. Expense Control: A Gamified, Semi-Automated, Crowd-Based Approach For Receipt Capturing. In Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI ’16). Association for Computing Machinery, New York, NY, USA, 31–42. in Google Scholar

[2] Fernando Andrade, Riichiro Mizoguchi, and Seiji Isotani. 2016. The Bright and Dark Sides of Gamification. In Intelligent Tutoring Systems, Alessandro Micarelli, John Stamper, and Kitty Panourgia (Eds.). Lecture Notes in Computer Science, Vol. 9684. Springer International Publishing, 1–11. in Google Scholar

[3] Gokhan Aydin. 2018. Effect of demographics on use intention of gamified systems. International Journal of Technology and Human Interaction (IJTHI) 14, 1, 1–21.10.4018/IJTHI.2018010101Search in Google Scholar

[4] Richard Bartle. 1996. Hearts, clubs, diamonds, spades: Players who suit MUDs. Journal of MUD research 1, 1, 19.Search in Google Scholar

[5] Martin Böckle, Isabel Micheel, Markus Bick, and Jasminko Novak. 2018. A Design Framework for Adaptive Gamification Applications. In Proceedings of the 51st Hawaii International Conference on System Sciences (Proceedings of the Annual Hawaii International Conference on System Sciences), Tung Bui (Ed.). Hawaii International Conference on System Sciences. in Google Scholar

[6] Bunchball. 2015. What are Game Mechanics? in Google Scholar

[7] Rachel C. Callan, Kristina N. Bauer, and Richard N. Landers. 2015. How to avoid the dark side of gamification: Ten business scenarios and their unintended consequences. In Gamification in education and business. Springer, 553–568.Search in Google Scholar

[8] Sebastian Deterding, Staffan Björk, Lennart Nacke, Dan Dixon, and Elizabeth Lawley. 2013. Designing gamification: creating gameful and playful experiences. In Extended Abstracts on Human Factors in Computing Systems. 3263–3266. in Google Scholar

[9] Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From Game Design Elements to Gamefulness: Defining Gamification. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (MindTrek ’11). ACM, 9–15. in Google Scholar

[10] Dominique Mangiatordi. 2018. Gamification at work: the 8 PLAYER TYPES. in Google Scholar

[11] Lauren Ferro, Steffen Walz, and Stefan Greuter. 2013. Towards personalised, gamified systems: An investigation into game design, personality and player typologies. In Proceedings of The 9th Australasian Conference on Interactive Entertainment Matters of Life and Death. in Google Scholar

[12] Google. 2020. Google Crowdsource. in Google Scholar

[13] Simone Hantke, Tobias Appel, and Björn Schuller. 2018. The Inclusion of Gamification Solutions to Enhance User Enjoyment on Crowdsourcing Platforms. In 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia). 1–6. in Google Scholar

[14] Eduardo Herranz, Ricardo Colomo-Palacios, and Antonio de Amescua Seco. 2015. Gamiware: A Gamification Platform for Software Process Improvement. In Systems, Software and Services Process Improvement, Rory V. O’Connor, Mariye Umay Akkaya, Kerem Kemaneci, Murat Yilmaz, Alexander Poth, and Richard Messnarz (Eds.). Springer International Publishing, Cham, 127–139.10.1007/978-3-319-24647-5_11Search in Google Scholar

[15] Karl Kapp. 2012. The gamification of learning and instruction: Game-based methods and strategies for training and education. Pfeiffer, San Francisco, CA.10.1145/2207270.2211316Search in Google Scholar

[16] Alireza Khakpour and Ricardo Colomo-Palacios. 2020. Convergence of Gamification and Machine Learning: A Systematic Literature Review. Technology, Knowledge and Learning. in Google Scholar

[17] Selay Arkün Kocadere and Şeyma Çağlar. 2018. Gamification from player type perspective: A case study. Journal of Educational Technology & Society 21, 3, 12–22.Search in Google Scholar

[18] Jonna Koivisto and Aqdas Malik. 2020. Gamification for Older Adults: A Systematic Literature Review. The Gerontologist. in Google Scholar PubMed PubMed Central

[19] Janaki Kumar and Mario Herger. 2013. Gamification at Work: Designing Engaging Business Software. In Design, User Experience, and Usability. Health, Learning, Playing, Cultural, and Cross-Cultural User Experience, Aaron Marcus (Ed.). Springer Berlin Heidelberg, 528–537.10.1007/978-3-642-39241-2_58Search in Google Scholar

[20] Alexandra L’Heureux, Katarina Grolinger, Wilson Akio Higashino, and Miriam A. M. Capretz. 2017. A Gamification Framework for Sensor Data Analytics. In 2017 IEEE International Congress on Internet of Things (ICIOT). 74–81. in Google Scholar

[21] Daria Lopukhina. 2018. How Gamification in the Workplace Impacts Employee Productivity. in Google Scholar

[22] Andrzej Marczewski. 2013. Game Mechanics in Gamification. in Google Scholar

[23] Andrzej Marczewski. 2013. The Intrinsic Motivation RAMP. in Google Scholar

[24] Andrzej Marczewski. 2015. Even ninja monkeys like to play: Gamification, game thinking & motivational design. Gamified UK. in Google Scholar

[25] Jane McGonigal. 2011. Reality is broken. Penguin Press. in Google Scholar

[26] Nina Runge, Dirk Wenig, Danny Zitzmann, and Rainer Malaka. 2015. Tags You Don’t Forget: Gamified Tagging of Personal Images. In Entertainment Computing – ICEC 2015, Konstantinos Chorianopoulos, Monica Divitini, Jannicke Baalsrud Hauge, Letizia Jaccheri, and Rainer Malaka (Eds.). Springer International Publishing, 301–314.10.1007/978-3-319-24589-8_23Search in Google Scholar

[27] Richard M. Ryan and Edward L. Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist 55, 1, 68–78. in Google Scholar

[28] Lorenzo Servadei, Rainer Schmidt, Christina Eidelloth, and Andreas Maier. 2018. In Medical Monkeys: A Crowdsourcing Approach to Medical Big Data. 87–97. in Google Scholar

[29] Samuel Suikkanen. 2019-06-17. Gamification in video labeling; Videoiden merkkaamisen pelillistäminen. G2 pro gradu, diplomityö. in Google Scholar

[30] Richard H. Thaler and Cass R. Sunstein. 2008. Nudge – Improving Decisions About Health, Wealth, and Happiness (rev. and expanded ed., with a new afterword and a new chapter ed.). Penguin. 304 Pages.Search in Google Scholar

[31] Rob van Roy, Sebastian Deterding, and Bieke Zaman. 2018. Uses and Gratifications of Initiating Use of Gamified Learning Platforms. In Asian CHI Symposium 2018, Eunice Sari (Ed.). CHI Conference, ACM, 1–6. in Google Scholar

[32] Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the 2004 conference on Human factors in computing systems – CHI ’04, Elizabeth Dykstra-Erickson and Manfred Tscheligi (Eds.). ACM Press, 319–326. in Google Scholar

[33] Lincoln Wood and Torsten Reiners. 2015. Gamification. In Encyclopedia of Information Science and Technology. 3039–3047. in Google Scholar

Published Online: 2021-04-22
Published in Print: 2021-04-27

© 2021 Walter de Gruyter GmbH, Berlin/Boston

Scroll Up Arrow