Accessible Requires Authentication Published by Oldenbourg Wissenschaftsverlag March 27, 2018

Multiscreen Patterns – Interactions Across the Borders of Devices

Dominick Madden, Horst Schneider and Kirstin Kohler
From the journal i-com


Design patterns are solutions for common design problems used in a number of fields including architecture, software development and user experience design. We compiled a pattern library for the usage of gesture-enabled interactions between different devices with screens, the so called multiscreen context. This library provides simple and intuitive gestures for connecting and disconnecting devices wirelessly as well as gestures for the exchange of data between these devices, like swiping a document from one’s tablet to the tablets of the surrounding colleagues in the same room. The library is the result of a 2.5 years running project in cooperation with three small and medium companies. We compiled the library by conducting an intense literature survey in parallel with several iterations of practical software projects in the multiscreen context. The literature survey inspired the practice-oriented project work. Being involved in software projects enabled us to identify challenges in the design and implementation of the gesture set. Therefore, we generated valuable insight for a comprehensive description of gestural interactions.

With our set of patterns, we aim to support interaction designers to choose the appropriate gesture for their given context. The patterns serve as inspiration by showing the different possibilities but also provide guidance how to design and implement a selected gesture for a given context. They help designing the details of an interaction by breaking it down into its smallest parts. To support the developers of these interactions our pattern descriptions are enriched with an Android library containing lifecycle events and the necessary gesture recognition logic. This paper provides an overview of the pattern library. In addition, the structure and the usage of the library is described in more detail with the means of one sample pattern. The pattern library is openly accessible.

Funding statement: The research was founded by the BMBF conducted in the project SysPlace (Grant number: 01IS14018D).


[1] Jan O. Borchers. 2000. A pattern approach to interaction design. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 369–378). ACM. Search in Google Scholar

[2] Matthew Chalmers and I. MacColl. 2003. Seamful and seamless design in ubiquitous computing. In Workshop at the crossroads: The interaction of HCI and systems issues in UbiComp, (Vol. 8). Search in Google Scholar

[3] Raimund Dachselt and R. Buchholz. 2009 April. Natural throw and tilt interaction between mobile phones and distant displays. In CHI’09 extended abstracts on Human factors in computing systems (pp. 3253–3258). ACM. Search in Google Scholar

[4] Erich Gamma, R. Helm, R. Johnson and J. Vlissides. 1995. Design pattern. Elements of Object Oriented Software. Addison Wesley, Reading, MA. Search in Google Scholar

[5] Frank Thomas and O. Johnston. 1981. The Illusion of Life: Disney Animation. Search in Google Scholar

[6] Google Inc. The New Multi-screen World: Understanding Cross-pllattform Consumer Behavior. 2012, (accessed 20. Dezember 2017). Search in Google Scholar

[7] Google Inc., Material Design Guidelines. (access 02. Januar 2018). Search in Google Scholar

[8] Ken Hinckley, G. Ramos, F. Guimbretiere, P. Baudisch and M. Smith. 2004 May. Stitching: pen gestures that span multiple displays. In Proceedings of the working conference on Advanced visual interfaces (pp. 23–31). ACM. Search in Google Scholar

[9] Lars Holmquist, F. Mattern, B. Schiele, P. Alahuhta, M. Beigl and H. W. Gellersen. 2001. Smart-its friends: A technique for users to easily establish connections between smart artefacts. In Ubicomp 2001: Ubiquitous Computing (pp. 116–122). Springer, Berlin/Heidelberg. Search in Google Scholar

[10] Jörn Hurtienne and J. H. Israel, 2007 February. Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. In Proceedings of the 1st international conference on Tangible and embedded interaction (pp. 127–134). ACM. Search in Google Scholar

[11] Manuel Imaz and D. Benyon. 2007. Designing with blends. MIT Press. Search in Google Scholar

[12] Hans-Christian Jetter, H. Reiterer and F. Geyer 2014. Blended Interaction: understanding natural human–computer interaction in post-WIMP interactive spaces. Personal and Ubiquitous Computing, 18(5), pp. 1139–1158.10.1007/s00779-013-0725-4 Search in Google Scholar

[13] Nicolai Marquardt and S. Greenberg. 2015. Proxemic interactions: From theory to practice. Synthesis Lectures on Human-Centered Informatics, 8(1), pp. 1–199.10.2200/S00619ED1V01Y201502HCI025 Search in Google Scholar

[14] Jun Rekimoto. 1996 November. Tilting operations for small screen interfaces. In Proceedings of the 9th annual ACM symposium on User interface software and technology (pp. 167–168). ACM. Search in Google Scholar

[15] Dan Saffer, D. 2013. Microinteractions: designing with details. O’Reilly Media, Inc. Search in Google Scholar

[16] Lucia Terrenghi, A. Quigley, A. Dix. 2009. A taxonomy for and analysis of multi-person-display ecosystems. Personal and Ubiquitous Computing 13(8), pp. 583–598.10.1007/s00779-009-0244-5 Search in Google Scholar

[17] Mark Weiser. 1994 March. Ubiquitous computing. In ACM Conference on Computer Science (p. 418). Search in Google Scholar

Published Online: 2018-03-27
Published in Print: 2018-04-25

© 2018 Walter de Gruyter GmbH, Berlin/Boston