Skip to content
Licensed Unlicensed Requires Authentication Published by Oldenbourg Wissenschaftsverlag April 22, 2021

Paper2Wire – A Case Study of User-Centred Development of Machine Learning Tools for UX Designers

Daniel Buschek ORCID logo, Charlotte Anlauff and Florian Lachner
From the journal i-com


This paper reflects on a case study of a user-centred concept development process for a Machine Learning (ML) based design tool, conducted at an industry partner. The resulting concept uses ML to match graphical user interface elements in sketches on paper to their digital counterparts to create consistent wireframes. A user study (N=20) with a working prototype shows that this concept is preferred by designers, compared to the previous manual procedure. Reflecting on our process and findings we discuss lessons learned for developing ML tools that respect practitioners’ needs and practices.

Article note

Florian Lachner is now at Google.

Funding statement: This project is funded by the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt).


We thank MaibornWolff for the support, for participating in interviews on the prototyping practice and for testing.


[1] Microsoft AI. 2018. Transform sketches into HTML using AI. in Google Scholar

[2] Michel Beaudouin-Lafon and Wendy Mackay. 2003. Prototyping tools and techniques. Human Computer Interaction-Development Process (2003), 122–142.Search in Google Scholar

[3] Tony Beltramelli. 2018. Pix2code: Generating Code from a Graphical User Interface Screenshot. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS ’18) (Paris, France). Association for Computing Machinery, New York, NY, USA, Article 3, 6 pages. in Google Scholar

[4] Susanne Bødker and Kaj Grønbæk. 1991. Cooperative prototyping: users and designers in mutual activity. International Journal of Man-Machine Studies 34, 3 (Mar 1991), 453–478. in Google Scholar

[5] Maureen Carroll, Shelley Goldman, Leticia Britos, Jaime Koh, Adam Royalty, and Michael Hornstein. 2010. Destination, imagination and the fires within: Design thinking in a middle school classroom. International Journal of Art & Design Education 29, 1 (Mar 2010), 37–53. in Google Scholar

[6] Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18) (Gothenburg, Sweden). ACM, New York, NY, USA, 665–676. in Google Scholar

[7] Juliet Corbin and Anselm Strauss. 2014. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications.Search in Google Scholar

[8] Sebastian Deterding, Jonathan Hook, Rebecca Fiebrink, Marco Gillies, Jeremy Gow, Memo Akten, Gillian Smith, Antonios Liapis, and Kate Compton. 2017. Mixed-Initiative Creative Interfaces. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17) (Denver, Colorado, USA). ACM, New York, NY, USA, 628–635. in Google Scholar

[9] Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems – CHI ’17. ACM Press, New York, New York, USA, 278–288. in Google Scholar

[10] Patrick Hebron. 2017. Rethinking Design Tools in the Age of Machine Learning. Medium. in Google Scholar

[11] Eric Horvitz. 1999. Principles of Mixed-initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’99) (Pittsburgh, Pennsylvania, USA). ACM, New York, NY, USA, 159–166. in Google Scholar

[12] Levent Burak Kara and Thomas F. Stahovich. 2005. An image-based, trainable symbol recognizer for hand-drawn sketches. Computers & Graphics 29, 4 (2005), 501–517.10.1016/j.cag.2005.05.004Search in Google Scholar

[13] Janin Koch. 2017. Design implications for Designing with a Collaborative AI. The AAAI 2017 Spring Symposium on Designing the User Experience of Machine Learning Systems Technical Report SS-17-04 Design (2017), 415–418.Search in Google Scholar

[14] James A. Landay and Brad A. Myers. 2001. Sketching interfaces: Toward more human interface design. Computer 34, 3 (2001), 56–64.10.1109/2.910894Search in Google Scholar

[15] Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and Evaluation of a User Experience Questionnaire. In HCI and Usability for Education and Work, Andreas Holzinger (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 63–76.10.1007/978-3-540-89350-9_6Search in Google Scholar

[16] Mark W. Newman and James A. Landay. 2000. Sitemaps, Storyboards, and Specifications: A Sketch of Web Site Design Practice. In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques. ACM, 263–274.10.1145/347642.347758Search in Google Scholar

[17] Jakob Nielsen and Heather Desurvire. 1993. Comparative Design Review: An Exercise in Parallel Design. In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems (CHI ’93) (Amsterdam, The Netherlands). ACM, New York, NY, USA, 414–417. in Google Scholar

[18] Antti Oulasvirta. 2017. User interface design with combinatorial optimization. Computer 50, 1, 40–47. in Google Scholar

[19] Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, Gregorio Palmas, and Tino Weinkauf. 2018. Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings (UIST ’18 Adjunct) (Berlin, Germany). ACM, New York, NY, USA, 16–19. in Google Scholar

[20] Juergen Sauer and Andreas Sonderegger. 2009. The influence of prototype fidelity and aesthetics of design in usability tests: Effects on user behaviour, subjective evaluation and emotion. Applied Ergonomics 40, 4, 670–677. in Google Scholar PubMed

[21] A. Schmidt. 2017. Technologies to amplify the mind. Computer 50, 10 102–106. in Google Scholar

[22] Ben Shneiderman. 1999. User Interfaces for Creativity Support Tools. In Proceedings of the 3rd Conference on Creativity & Cognition (C&C ’99) (Loughborough, United Kingdom). ACM, New York, NY, USA, 15–22. in Google Scholar

[23] Herbert A. Simon. 1996. The Sciences of the Artificial. MIT Press.Search in Google Scholar

[24] Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Andrew J. Ko. 2018. Rewire: Interface Design Assistance from Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18) (Montreal QC, Canada). ACM, New York, NY, USA, Article 504, 12 pages. in Google Scholar

[25] Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and Explore with a Layout Optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS ’16) (Brisbane, QLD, Australia). ACM, New York, NY, USA, 543–555. in Google Scholar

[26] Miriam Walker, Leila Takayama, and James A. Landay. 2002. High-fidelity or Low-fidelity, Paper or Computer? Choosing Attributes when Testing Web Prototypes. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 46. SAGE Publications, Los Angeles, CA, 661–665.Search in Google Scholar

[27] Benjamin Wilkins. 2018. Sketching Interfaces. Airbnb.Design. in Google Scholar

[28] Qian Yang. 2017. The Role of Design in Creating Machine-Learning-Enhanced User Experience. The AAAI 2017 Spring Symposium on Designing the User Experience of Machine Learning Systems, Technical Report SS-17-04 (2017), 406–411. in Google Scholar

[29] Qian Yang, Nikola Banovic, and John Zimmerman. 2018. Mapping Machine Learning Advances from HCI Research to Reveal Starting Places for Design Innovation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 130.Search in Google Scholar

Published Online: 2021-04-22
Published in Print: 2021-04-27

© 2021 Walter de Gruyter GmbH, Berlin/Boston

Scroll Up Arrow