Skip to content
Licensed Unlicensed Requires Authentication Published by Oldenbourg Wissenschaftsverlag January 15, 2021

How to Handle Health-Related Small Imbalanced Data in Machine Learning?

  • Maria Rauschenberger

    Maria Rauschenberger is Professor for Digital Media at the University of Applied Science in Emden/Leer. Before she was a Post-Doc at the Max-Planck Institute for Software Systems in Saarbrücken, research associate at the OFFIS – Institute for Information Technology in Oldenburg and Product Owner at MSP Medien Systempartner in Bremen/Oldenburg. Maria did her Ph.D. at Universitat Pompeu Fabra in the Department of Information and Communication Technologies under the supervision of Luz Rello and Ricardo Baeza-Yates since early 2016, graduating in 2019 with the highest outcome: Excellent Cum Laude. Her thesis focused on the design of a language-independent content game for early detection of children with dyslexia. She mastered the challenges of user data collection as well as of small data analysis for interaction data using machine learning and shows innovative and solid approaches. Such a tool will help children with dyslexia to overcome their future reading and writing problems by early screening. All this work has been awarded every year (three years in a row) in Germany with a special scholarship (fem:talent) as well as with the prestigious German Reading 2017 award and recently with the 2nd place of the Helene-Lange-Preis.

    ORCID logo EMAIL logo
    and Ricardo Baeza-Yates

    Ricardo Baeza-Yates is since 2017 the Director of Data Science Programs at Northeastern University, Silicon Valley campus, and part-time professor at University Pompeu Fabra in Barcelona, Spain; as well as at University of Chile. Before, he was VP of Research at Yahoo Labs, based in Barcelona, Spain, and later in Sunnyvale, California, from 2006 to 2016. He is co-author of the best-seller Modern Information Retrieval textbook published by Addison-Wesley in 1999 and 2011 (2nd edition), that won the ASIST 2012 Book of the Year award. From 2002 to 2004 he was elected to the Board of Governors of the IEEE Computer Society and between 2012 and 2016 was elected for the ACM Council. In 2009 he was named ACM Fellow and in 2011 IEEE Fellow, among other awards and distinctions. Finally, in 2018 he obtained the National Spanish Award for Applied Research in Computing. He obtained a Ph.D. in CS from the University of Waterloo, Canada, in 1989, and his areas of expertise are web search and data mining, information retrieval, data science and algorithms in general. He has over 40 thousand citations in Google Scholar with an h-index of over 80.

    ORCID logo
From the journal i-com

Abstract

When discussing interpretable machine learning results, researchers need to compare them and check for reliability, especially for health-related data. The reason is the negative impact of wrong results on a person, such as in wrong prediction of cancer, incorrect assessment of the COVID-19 pandemic situation, or missing early screening of dyslexia. Often only small data exists for these complex interdisciplinary research projects. Hence, it is essential that this type of research understands different methodologies and mindsets such as the Design Science Methodology, Human-Centered Design or Data Science approaches to ensure interpretable and reliable results. Therefore, we present various recommendations and design considerations for experiments that help to avoid over-fitting and biased interpretation of results when having small imbalanced data related to health. We also present two very different use cases: early screening of dyslexia and event prediction in multiple sclerosis.

ACM CCS:

About the authors

Maria Rauschenberger

Maria Rauschenberger is Professor for Digital Media at the University of Applied Science in Emden/Leer. Before she was a Post-Doc at the Max-Planck Institute for Software Systems in Saarbrücken, research associate at the OFFIS – Institute for Information Technology in Oldenburg and Product Owner at MSP Medien Systempartner in Bremen/Oldenburg. Maria did her Ph.D. at Universitat Pompeu Fabra in the Department of Information and Communication Technologies under the supervision of Luz Rello and Ricardo Baeza-Yates since early 2016, graduating in 2019 with the highest outcome: Excellent Cum Laude. Her thesis focused on the design of a language-independent content game for early detection of children with dyslexia. She mastered the challenges of user data collection as well as of small data analysis for interaction data using machine learning and shows innovative and solid approaches. Such a tool will help children with dyslexia to overcome their future reading and writing problems by early screening. All this work has been awarded every year (three years in a row) in Germany with a special scholarship (fem:talent) as well as with the prestigious German Reading 2017 award and recently with the 2nd place of the Helene-Lange-Preis.

Ricardo Baeza-Yates

Ricardo Baeza-Yates is since 2017 the Director of Data Science Programs at Northeastern University, Silicon Valley campus, and part-time professor at University Pompeu Fabra in Barcelona, Spain; as well as at University of Chile. Before, he was VP of Research at Yahoo Labs, based in Barcelona, Spain, and later in Sunnyvale, California, from 2006 to 2016. He is co-author of the best-seller Modern Information Retrieval textbook published by Addison-Wesley in 1999 and 2011 (2nd edition), that won the ASIST 2012 Book of the Year award. From 2002 to 2004 he was elected to the Board of Governors of the IEEE Computer Society and between 2012 and 2016 was elected for the ACM Council. In 2009 he was named ACM Fellow and in 2011 IEEE Fellow, among other awards and distinctions. Finally, in 2018 he obtained the National Spanish Award for Applied Research in Computing. He obtained a Ph.D. in CS from the University of Waterloo, Canada, in 1989, and his areas of expertise are web search and data mining, information retrieval, data science and algorithms in general. He has over 40 thousand citations in Google Scholar with an h-index of over 80.

References

[1] Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan Kankanhalli. 2018. Trends and Trajectories for Explainable, Accountable and Intelligible Systems. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems – CHI’18. ACM Press, New York, New York, USA, 1–18. https://doi.org/10.1145/3173574.3174156.10.1145/3173574.3174156Search in Google Scholar

[2] Muneeb Imtiaz Ahmad and Suleman Shahid. 2015. Design and Evaluation of Mobile Learning Applications for Autistic Children in Pakistan. In INTERACT (Lecture Notes in Computer Science, Vol. 9296), Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing, Cham, 436–444. https://doi.org/10.1007/978-3-319-22701-6.10.1007/978-3-319-22701-6Search in Google Scholar

[3] Jonathan Arnowitz, Michael Arent, and Nevin Berger. 2007. Effective Prototyping for Software Makers. Morgan Kaufmann, unknown. 584 pages. https://www.oreilly.com/library/view/effective-prototypingfor/9780120885688/.10.1016/B978-012088568-8/50003-5Search in Google Scholar

[4] Ricardo Baeza-Yates. Bias on the web. Commun. ACM 61, 6 (may 2018), 54–61. https://doi.org/10.1145/3209581.10.1145/3209581Search in Google Scholar

[5] Ricardo Baeza-Yates. 2018. BIG, small or Right Data: Which is the proper focus? https://www.kdnuggets.com/2018/10/big-small-right-data.html [Online, accessed 22-July-2019].Search in Google Scholar

[6] Protima Banerjee. 2004. About Face 2.0: The Essentials of Interaction Design. Vol. 3. Wiley Publishing, Inc., USA, 223–225. https://doi.org/10.1057/palgrave.ivs.9500066.10.1057/palgrave.ivs.9500066Search in Google Scholar

[7] Gerd Berget and Andrew MacFarlane. 2019. Experimental Methods in IIR. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval – CHIIR’19. ACM Press, New York, New York, USA, 93–101. https://doi.org/10.1145/3295750.3298939.10.1145/3295750.3298939Search in Google Scholar

[8] Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Vol. 1. Springer Science+Business Media, LLC, Singapore, 1–738. http://cds.cern.ch/record/998831/files/9780387310732_TOC.pdf.Search in Google Scholar

[9] Donald W. Black, Jon E. Grant, and American Psychiatric Association. 2016. DSM-5 guidebook: The essential companion to the Diagnostic and statistical manual of mental disorders, fifth edition (5th edition ed.). American Psychiatric Association, London. 543 pages. https://www.appi.org/dsm-5_guidebook.Search in Google Scholar

[10] S. Bos, I. Brorson, E.A. Hogestol, J. Saez-Rodriguez, A. Uccelli, F. Paul, P. Villoslada, H.F. Harbo, and T. Berge. Sys4MS: Multiple sclerosis genetic burden score in a systems biology study of MS patients from four countries. European Journal of Neurology 26 (2019), 159.Search in Google Scholar

[11] Henning Brau and Florian Sarodnick. 2006. Methoden der Usability Evaluation (Methods of Usability Evaluation) (2. ed.). Verlag Hans Huber, Bern. 251 pages. http://d-nb.info/1003981860, http://www.amazon.com/Methoden-Usability-Evaluation-Henning-Brau/dp/3456842007.Search in Google Scholar

[12] Kelly Caine. 2016. Local Standards for Sample Size at CHI. In CHI’16. ACM, San Jose California, USA, 981–992. https://doi.org/10.1145/2858036.2858498.10.1145/2858036.2858498Search in Google Scholar

[13] André M. Carrington, Paul W. Fieguth, Hammad Qazi, Andreas Holzinger, Helen H. Chen, Franz Mayr, and Douglas G. Manuel. A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Medical Informatics and Decision Making 20, 1 (2020), 1–12. https://doi.org/10.1186/s12911-019-1014-6.10.1186/s12911-019-1014-6Search in Google Scholar PubMed PubMed Central

[14] Greig De Zubicaray and Niels Olaf Schiller. 2018. The Oxford handbook of neurolinguistics. Oxford University Press, New York, NY. https://www.worldcat.org/title/oxford-handbook-ofneurolinguistics/oclc/1043957419&referer=brief_results.Search in Google Scholar

[15] Tom Dietterich. Overfitting and undercomputing in machine learning. Comput. Surveys 27, 3 (sep 1995), 326–327. https://doi.org/10.1145/212094.212114.10.1145/212094.212114Search in Google Scholar

[16] Julian J. Faraway and Nicole H. Augustin. When small data beats big data. Statistics & Probability Letters 136 (may 2018), 142–145. https://doi.org/10.1016/j.spl.2018.02.031.10.1016/j.spl.2018.02.031Search in Google Scholar

[17] Andy P. Field and Graham Hole. 2003. How to design and report experiments. SAGE Publications, London. 384 pages.Search in Google Scholar

[18] Ana Freire, Magi Andorra, Irati Zubizarreta, Nicole Kerlero de Rosbo, Steffan R. Bos, Melanie Rinas, Einar A. Høgestøl, Sigrid A. de Rodez Benavent, Tone Berge, Priscilla Bäcker-Koduah, Federico Ivaldi, Maria Cellerino, Matteo Pardini, Gemma Vila, Irene Pulido-Valdeolivas, Elena H. Martinez-Lapiscina, Alex Brandt, Julio Saez-Rodriguez, Friedemann Paul, Hanne F. Harbo, Antonio Uccelli, Ricardo Baeza-Yates, and Pablo Villoslada. to appear. Precision medicine in MS: a multi-omics, imaging, and machine learning approach to predict disease severity.Search in Google Scholar

[19] Koichi Fujiwara, Yukun Huang, Kentaro Hori, Kenichi Nishioji, Masao Kobayashi, Mai Kamaguchi, and Manabu Kano. Over- and Under-sampling Approach for Extremely Imbalanced and Small Minority Data Problem in Health Record Analysis. Frontiers in Public Health 8 (may 2020), 178. https://doi.org/10.3389/fpubh.2020.00178.10.3389/fpubh.2020.00178Search in Google Scholar PubMed PubMed Central

[20] Ombretta Gaggi, Giorgia Galiazzo, Claudio Palazzi, Andrea Facoetti, and Sandro Franceschini. 2012. A serious game for predicting the risk of developmental dyslexia in pre-readers children. In 2012 21st International Conference on Computer Communications and Networks, ICCCN 2012 – Proceedings. IEEE, Munich, Germany, 1–5. https://doi.org/10.1109/ICCCN.2012.6289249.10.1109/ICCCN.2012.6289249Search in Google Scholar

[21] Martin Grund, Carl Ludwig Naumann, and Gerhard Haug. 2004. Diagnostischer Rechtschreibtest für 5. Klassen: DRT 5 (Diagnostic spelling test for fifth grade: DRT 5) (2., aktual ed.). Beltz Test, Göttingen. https://www.testzentrale.de/shop/diagnostischer-rechtschreibtest-fuer-5-klassen.html.Search in Google Scholar

[22] Alan Hevner, Salvatore T. March, Jinsoo Park, and Sudha Ram. Design Science in Information Systems Research. MIS Quarterly 28, 1 (2004), 75. https://doi.org/10.2307/25148625.10.1007/978-1-4419-5653-8_2Search in Google Scholar

[23] Andreas Hinderks, Martin Schrepp, Maria Rauschenberger, Siegfried Olschner, and Jörg Thomaschewski. 2012. Konstruktion eines Fragebogens für jugendliche Personen zur Messung der User Experience (Construction of a questionnaire for young people to measure user experience). In Usability Professionals Konferenz 2012. German UPA e.V., Stuttgart, UPA, Stuttgart, 78–83.Search in Google Scholar

[24] Steven A. Hoozemans. 2020. Machine Learning with care: Introducing a Machine Learning Project Method. 129 pages. https://repository.tudelft.nl/islandora/object/uuid:6be8ea7b-2a87-45d9-aaa8-c82ff28d56c2.Search in Google Scholar

[25] Robert R. Huffman, Axel Roesler, and Brian M. Moon. What is design in the context of human-centered computing? IEEE Intelligent Systems 19, 4 (2004), 89–95. https://doi.org/10.1109/MIS.2004.36.10.1109/MIS.2004.36Search in Google Scholar

[26] ISO/TC 159/SC 4 Ergonomics of human-system interaction. 2010. Part 210: Human-centred design for interactive systems. In Ergonomics of human-system interaction. Vol. 1. International Organization for Standardization (ISO), Brussels, 32. https://www.iso.org/standard/52075.html.Search in Google Scholar

[27] ISO/TC 159/SC 4 Ergonomics of human-system interaction. 2018. ISO 9241-11, Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. 2018 pages. https://www.iso.org/standard/63500.html, https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en.Search in Google Scholar

[28] Anil Jain and Douglas Zongker. Feature selection: evaluation, application, and small sample performance. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 2 (1997), 153–158. https://doi.org/10.1109/34.574797.10.1109/34.574797Search in Google Scholar

[29] Anuradha Kar. MLGaze: Machine Learning-Based Analysis of Gaze Error Patterns in Consumer Eye Tracking Systems. Vision (Switzerland) 4, 2 (may 2020), 1–34. https://doi.org/10.3390/vision4020025, arXiv:2005.03795.10.3390/vision4020025Search in Google Scholar PubMed PubMed Central

[30] Jakob Nielsen. Why You Only Need to Test with 5 Users. Jakob Nielsens Alertbox 19 (sep 2000), 1–4. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/, http://www.useit.com/alertbox/20000319.html [Online, accessed 11-July-2019].Search in Google Scholar

[31] Ken Peffers, Tuure Tuunanen, Marcus A. Rothenberger, and Samir Chatterjee. A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems 24, 8 (2007), 45–78. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.535.7773&rep=rep1&type=pdf.10.2753/MIS0742-1222240302Search in Google Scholar

[32] Maria Rauschenberger. 2019. Early screening of dyslexia using a languageindependent content game and machine learning. Ph.D. Dissertation. Universitat Pompeu Fabra. https://doi.org/10.13140/RG.2.2.27740.95363.Search in Google Scholar

[33] Maria Rauschenberger and Ricardo Baeza-Yates. 2020. Recommendations to Handle Health-related Small Imbalanced Data in Machine Learning. In Mensch und Computer 2020 – Workshopband (Human and Computer 2020 – Workshop proceedings), Bernhard Christian Hansen and Nürnberger Andreas Preim (Ed.). Gesellschaft für Informatik e.V., Bonn, 1–7. https://doi.org/10.18420/muc2020-ws111-333.Search in Google Scholar

[34] Maria Rauschenberger, Ricardo Baeza-Yates, and Luz Rello. 2020. Screening Risk of Dyslexia through a Web-Game using Language-Independent Content and Machine Learning. In W4a’2020. ACM Press, Taipei, 1–12. https://doi.org/10.1145/3371300.3383342.10.1145/3371300.3383342Search in Google Scholar

[35] Maria Rauschenberger, Silke Füchsel, Luz Rello, Clara Bayarri, and Jörg Thomaschewski. 2015. Exercises for German-Speaking Children with Dyslexia. In Human-Computer Interaction – INTERACT 2015. Springer, Bamberg, Germany, 445–452.10.1007/978-3-319-22701-6_33Search in Google Scholar

[36] Maria Rauschenberger, Christian Lins, Noelle Rousselle, Sebastian Fudickar, and Andreas Hain. 2019. A Tablet Puzzle to Target Dyslexia Screening in Pre-Readers. In Proceedings of the 5th EAI International Conference on Smart Objects and Technologies for Social Good – GOODTECHS. ACM, Valencia, 155–159.10.1145/3342428.3342679Search in Google Scholar

[37] Maria Rauschenberger, Siegfried Olschner, Manuel Perez Cota, Martin Schrepp, and Jörg Thomaschewski. 2012. Measurement of user experience: A Spanish Language Version of the User Experience Questionnaire (UEQ). In Sistemas Y Tecnologias De Informacion, A. Rocha, J.A. CalvoManzano, L.P. Reis, and M.P. Cota (Eds.). IEEE, Madrid, Spain, 471–476.Search in Google Scholar

[38] Maria Rauschenberger, Luz Rello, and Ricardo Baeza-Yates. 2019. Technologies for Dyslexia. In Web Accessibility Book (2nd ed.), Yeliz Yesilada and Simon Harper (Eds.). Vol. 1. Springer-Verlag London, London, 603–627. https://doi.org/10.1007/978-1-4471-7440-0.10.1007/978-1-4471-7440-0_31Search in Google Scholar

[39] Maria Rauschenberger, Luz Rello, Ricardo Baeza-Yates, and Jeffrey P. Bigham. 2018. Towards language independent detection of dyslexia with a web-based game. In W4A’18: The Internet of Accessible Things. ACM, Lyon, France, 4–6. https://doi.org/10.1145/3192714.3192816.10.1145/3192714.3192816Search in Google Scholar

[40] Maria Rauschenberger, Martin Schrepp, Manuel Perez Cota, Siegfried Olschner, and Jörg Thomaschewski. Efficient Measurement of the User Experience of Interactive Products. How to use the User Experience Questionnaire (UEQ). Example: Spanish Language. International Journal of Artificial Intelligence and Interactive Multimedia (IJIMAI) 2, 1 (2013), 39–45. http://www.ijimai.org/journal/sites/default/files/files/2013/03/ijimai20132_15_pdf_35685.pdf.10.9781/ijimai.2013.215Search in Google Scholar

[41] Maria Rauschenberger, Martin Schrepp, and Jörg Thomaschewski. 2013. User Experience mit Fragebögen messen – Durchführung und Auswertung am Beispiel des UEQ (Measuring User Experience with Questionnaires–Execution and Evaluation using the Example of the UEQ). In Usability Professionals Konferenz 2013. German UPA eV, Bremen, 72–76.Search in Google Scholar

[42] Maria Rauschenberger, Andreas Willems, Menno Ternieden, and Jörg Thomaschewski. Towards the use of gamification frameworks in learning environments. Journal of Interactive Learning Research 30, 2 (2019), 147–165. https://www.aace.org/pubs/jilr/, http://www.learntechlib.org/c/JILR/.Search in Google Scholar

[43] Luz Rello and Ricardo Baeza-Yates. 2013. Good fonts for dyslexia. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS’13). ACM, New York, NY, USA, 14. https://doi.org/10.1145/2513383.2513447.10.1145/2513383.2513447Search in Google Scholar

[44] Luz Rello, Enrique Romero, Maria Rauschenberger, Abdullah Ali, Kristin Williams, Jeffrey P. Bigham, and Nancy Cushen White. 2018. Screening Dyslexia for English Using HCI Measures and Machine Learning. In Proceedings of the 2018 International Conference on Digital Health – DH’18. ACM Press, New York, New York, USA, 80–84. https://doi.org/10.1145/3194658.3194675.10.1145/3194658.3194675Search in Google Scholar

[45] Claire Rowland and Martin Charlier. 2015. User Experience Design for the Internet of Things. O’Reilly Media, Inc., Boston, 1–37.Search in Google Scholar

[46] Scikit-learn. 2019. 3.1. Cross-validation: evaluating estimator performance. https://scikit-learn.org/stable/modules/cross_validation.html [Online, accessed 17-June-2019].Search in Google Scholar

[47] Scikit-learn. 2019. 3.3. Model evaluation: quantifying the quality of predictions. https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter [Online, accessed 23-July-2019].Search in Google Scholar

[48] Scikit-learn Developers. 2019. Scikit-learn Documentation. https://scikit-learn.org/stable/documentation.html [Online, accessed 20-June-2019].Search in Google Scholar

[49] Herbert A. Simon. 1997. The sciences of the artificial, (third edition). Vol. 3. MIT Press, London, England. 130 pages. https://doi.org/10.1016/S0898-1221(97)82941-0.10.1016/S0898-1221(97)82941-0Search in Google Scholar

[50] Claudia Steinbrink and Thomas Lachmann. 2014. Lese-Rechtschreibstörung (Dyslexia). Springer Berlin Heidelberg, Berlin. https://doi.org/10.1007/978-3-642-41842-6.10.1007/978-3-642-41842-6Search in Google Scholar

[51] Lieven Van den Audenaeren, Véronique Celis, Vero Van den Abeele, Luc Geurts, Jelle Husson, Pol Ghesquière, Jan Wouters, Leen Loyez, and Ann Goeleven. 2013. DYSL-X: Design of a tablet game for early risk detection of dyslexia in preschoolers. In Games for Health. Springer Fachmedien Wiesbaden, Wiesbaden, 257–266. https://doi.org/10.1007/978-3-658-02897-8_20.10.1007/978-3-658-02897-8_20Search in Google Scholar

[52] Sudhir Varma and Richard Simon. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics 7 (feb 2006), 91. https://doi.org/10.1186/1471-2105-7-91.10.1186/1471-2105-7-91Search in Google Scholar PubMed PubMed Central

[53] Torben Wallbaum, Maria Rauschenberger, Janko Timmermann, Wilko Heuten, and Susanne C.J. Boll. 2018. Exploring Social Awareness. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems – CHI’18. ACM Press, New York, New York, USA, 1–10. https://doi.org/10.1145/3170427.3174365.10.1145/3170427.3174365Search in Google Scholar

[54] Joseph G. Walls, George R. Widmeyer, and Omar A. El Sawy. Building an information system design theory for vigilant EIS. Information Systems Research 3, 1 (1992), 36–59. https://doi.org/10.1287/isre.3.1.36.10.1287/isre.3.1.36Search in Google Scholar

[55] Danding Wang, Qian Yang, Ashraf Abdul, Brian Y. Lim, and United States. 2019. Designing Theory-Driven User-Centric Explainable AI. In CHI’19. ACM, Glasgow, Scotland, UK, 1–15.Search in Google Scholar

[56] Huaxiu Yao, Xiaowei Jia, Vipin Kumar, and Zhenhui Li. 2020. Learning with Small Data, 3539–3540. https://doi.org/10.1145/3394486.3406466, arXiv:1910.00201.10.1145/3394486.3406466Search in Google Scholar

[57] I. Zubizarreta, F. Ivaldi, M. Rinas, E. Hogestol, S. Bos, T. Berge, P. Koduah, M. Cellerino, M. Pardini, G. Vila, et al. The Sys4MS project: personalizing health care in multiple sclerosis using systems medicine tools. Multiple Sclerosis Journal 24 (2018), 459.10.26226/morressier.5b7578645aff7400151f3456Search in Google Scholar

Published Online: 2021-01-15
Published in Print: 2021-01-26

© 2020 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 9.2.2023 from https://www.degruyter.com/document/doi/10.1515/icom-2020-0018/html
Scroll Up Arrow