Skip to content
Publicly Available Published by Oldenbourg Wissenschaftsverlag November 27, 2021

Changing Things so (Almost) Everything Stays the Same

Technical Challenges and Solutions in a Mixed-Reality System for Financial Services

  • Mateusz Dolata

    Dr. Mateusz Dolata, born 1988, is a full-time postdoctoral researcher at the University of Zurich, Department of Informatics. His research interests span co-located collaboration in IT-supported settings and application of artificial intelligence for the common good. He applies a multidisciplinary perspective shaped by his background in computational linguistics, philosophy, and applied computer science. He has co-authored numerous conference and journal articles which open the black box of human work practices in human-computer assemblages. His research has appeared in journals and proceeding series including Computer Supported Cooperative Work, Proceedings of the ACM on Human-Computer Interaction, Information Technology & People, or Business & Information Systems Engineering.

    ORCID logo EMAIL logo
    , Simon Schubiger

    Dr. Simon Schubiger works as a senior principal software engineer on 3D technology at Esri. Before, he was professor for computer graphics, teaching game design and audio/video processing at the University of Applied Sciences Northwestern Switzerland and was the head of software development at Esri R&D center in Zurich, Switzerland. He is co-founder of the ETH spin-off company Procedural Inc, acquired 2011 by Esri. Previously, he was lecturing mobile system architectures at ETH Zurich, worked for Swisscom Innovations, and as an associate researcher in the Pervasive and Artificial Intelligence group at the University of Fribourg. His research interests include 3D computer graphics, multimedia performance systems, mobile computing, knowledge representation, programming languages, and user interface design.

    , Doris Agotai

    Dr. Doris Agotai has headed the newly founded Institute for Interactive Technologies FHNW since 2018, which is a multidisciplinary informatics institute specializing in the development of digital interfaces for people and processes. Until the end of 2017, she was deputy head of the Institute for 4D Technologies and, as the person in charge of the Design & Technology research area, built up and established the focus on design in computer science. In this role, she realized numerous projects in applied research and development and, in addition to regular teaching engagements, accompanied student projects, including the interdisciplinary and international iPOLE project. Her research interests embrace design and deployment of bleeding-edge technologies in collaborative settings.

    and Gerhard Schwabe

    Dr. Gerhard Schwabe has been a full professor at the University of Zurich since 2002. He has studied collaboration at the granularity of dyads, small teams, large teams, organizations, communities and social networks. In doing so, he follows either an engineering approach (“design science”) or an exploratory approach – frequently in collaboration with companies and public organizations. He has published in computer science conferences as well as in major information systems conferences and journals. Currently, his research interests focus on blockchain applications and human-robot collaboration.

From the journal i-com


The deployment of mixed reality systems in professional settings demands adaptation of the physical environment and practices. However, technology-driven changes to the environment are problematic in some contexts. Specifically, face-to-face advisory services rely on scripted material routines using specific tools. This manuscript explores challenges encountered during the development of LivePaper, a mixed-reality system for supporting financial advisory services. First, the article presents a range of design requirements derived from existing literature and multiple years of research experience concerning advisory services and physical collaborative environments. Second, it discusses technical and design challenges that emerged when building LivePaper along with those requirements. Third, the article describes a range of technical solutions and new design ideas implemented in a working system to mitigate the encountered problems. It explores potential alternative solutions and delivers empirical or conceptual arguments for the choices made. The manuscript concludes with implications for the advisory services, the systems used to support such encounters, and specific technical guidance for the developers of mixed reality solutions in institutional settings. Overall, the article advances the discourse on the application of technology in advisory services, the use of mixed-reality systems in professional environments, and the physical nature of collaboration.

1 Introduction

Various mixed reality (MR) approaches diffuse into professional collaboration contexts across domains. They support teachers [48], gun violence prevention workers [70], or retail employees [5]. While the employed techniques and technology are improving, the proposed solutions imply changes to the physical environment of interaction and the embodied work practices. In some contexts, changing the environment can positively affect turning a textbook-based lesson into a multimodal augmented reality (AR) experience can contribute to the learning effects [48]. However, many professional settings are tightly related to a set of embodied practices in a specific environment. In such collaborative scenarios, making changes typical for MR, e. g., adapting the lighting conditions for spatial AR or equipping the participants with new virtual reality devices, generates an incoherent experience with expectations or obligations. We claim that financial advisory services belong to this category – they are so much defined by their scenery (a clean and bright room, neat and clean-cut personnel, table as an area of interaction) that any changes may yield an insincere or trivial picture. This makes developing MR solutions for such settings a challenge for designers and technicians. This manuscript tackles this issue by discussing technical difficulties in a project where the authors designed and developed an MR solution for mortgage advisory services. We call the solution LivePaper. It derives the design requirements by analyzing the current physical configuration of those encounters, points out the challenges that occurred when developing the envisioned system, presents alternatives, describes the implemented solutions to answer those challenges, and discusses their impact on the advisory service practice. This illustrates how to design MR for application in professional services.

Advisory services are ritualized physical encounters. The performance is adequate and trouble-free only in a specific physical environment, which allows for expected bodily interactions and involves typical material. In other words, designing material, environment, or bodily formations for advisory services needs to obey the expectations and obligations following the scripted rituals. Providing new practices and affording them with digital tools includes the risk of destroying the ritual-physical symbolic interaction. This challenges the designers of support systems who intend to improve the advisory services and integrate them into the digital landscape of service provision organizations: How to augment existing practices with the power of IT while keeping their symbolic character? How to embrace bodily and sensory symbolic interaction in new intended behaviors? How to keep the overall physical environment stable and in a form that fits the service provider’s image?

Mixed reality approaches have claimed to provide a way to augment analog practices with the digital processing of information. However, many available solutions essentially change the physical environment and require changes to the material used in interaction. Among others, spatial augmented reality systems are spreading widely in museums and art exhibitions – they use projection of virtual overlay on the objects, such that, in principle, the physical and material world remains untouched. However, they mainly enable a limited set of actions (turning a book’s page or clicking). Furthermore, in most cases, the system is placed in a well-designed environment where the system’s designer can influence the light conditions, the furniture, and the surfaces. However, advisory service environments and practices impose specific needs that are not ideal for spatial AR solutions: bright light or the possibility to reconfigure the interaction space easily based on the course of the collaboration. Without those features, the AR solution will likely interfere with the advisory service rituals and eventually negatively affect the bank or the advisor. Mixed reality approaches like spatial AR provide only a partial answer to the overall goal of digitally supporting advisory services and generates technical challenges so far unaddressed in the literature.

We identified a range of technical challenges during the development of a spatial AR system for bank advisory services. They result from the intention to keep the existing configuration and to afford compatible but, at the same time, digitally empowered practices. This manuscript describes the challenges, outlines possible alternative solutions, and provides the empirical or theoretical rationale behind the finally implemented approaches to answer the following research question: How to design spatial augmented-reality system for advisory services in financial institutions? The manuscript offers design guidance to spatial AR researchers concerning a range of implementation problems. Furthermore, deriving the design challenges from the advisory service scenario and the relevant literature on visual and sensory processing in humans extends the understanding of service encounters. It opens the area of advisory services up for further research in the human-computer interaction domain.

Other publications [9], [10], [21] feature the system described in the current article. Whereas they focus on the effects of the system on the perception of advisory services or the practices that emerge during the advisory service, this manuscript focuses on the design of the system and the technical challenges and solutions that overcome those challenges. This manuscript provides the community with much-needed guidance on how systems like LivePaper can be implemented in practice and generates an opportunity to transfer this technology and design knowledge to other application areas. Specifically, Challenges and Solutions, Discussion, and significant parts of Related Work are novel, previously unpublished content.

2 Related Work

2.1 Advisory Services as Rituals

To design adequate support for advisory services, one needs to understand what an advisory service is. Researchers have approached advisory services from numerous areas. Management scientists study them as dyadic decision situations – they aim at enhancing the overall quality of the final advice or decision through process innovation and expertise enhancement [45], [64]. Marketing research sees advisory services as a customer relationship channel. It increasingly advocates strategies oriented at long-term co-creative and mutually profitable relation rather than a one-time sale – it challenges the status quo and proposes new communication and listening routines to make the service appealing to the client [56], [68], [72], [82]. Literature concerned with supporting collaborative work frames advisory services as work – it approaches various collaborative practices and tries to improve the overall service by providing digital support for the existing or new practices [20], [27], [31]. Finally, sociologists and linguists see advisory services as a form of institutional talk and investigate how the sequential character of the services corresponds to the institutional identities of the participants – they contribute an understanding of the fundamental conversation processes in various advisory service or selling situations [37], [66], [81]. The latter perspective includes recent works on the material aspects of the interaction between advisees and advisors [19], [81]. Research offers multiple entry points to the study of advisory services and applies various lenses to understand and improve them.

We conceptualize advisory services as, simultaneously, physical, and ritual encounters between two individuals, an advisor and an advisee. This framing corresponds strongly to the institutional talk perspective [37], [66], [81] and the material lens [19], [81]. Institutional talk discourse outgrows, among others, Goffman’s research on rituals in encounters between humans [24]. Through explicitly referring to rituals, we want to stress that the performances during advisory services have a recurring, repeatable character and, at the same time, celebrate a specific order in which the participants act to establish or protect their reputation or, in Goffman’s terms, their face. For instance, the advisor protects her identity as a domain expert, a representative of an institution, and a customer-centric frontline employee [20].

On the other hand, by emphasizing the physical character, we point out that the space, human bodies, furniture, and materials belong to the ritual [22]. The design of the room in which the advisory takes place contributes to the typical advisory service experience and embodied interaction with the advisor. In advisory services as physical rituals, humans act according to obligations to others and expectations one has of others while employing symbolic acts, which involve material and space [38]. In particular, advisors and advisees distribute documents in space to illustrate complex matters and to exercise control over interaction space, use expressive gestures to introduce oneself, employ posture to signalize attention or disinterest, and deliberately take care of the surroundings to prepare an advisory service [19]. For instance, when presenting a business card to the client in a dignified manner, the advisor employs a range of verbal, bodily, and material means to express oneself and the institution [19]. In mundane actions, the physical and ritual nature of advisory services comes to the surface. Designing a system for advisory services requires careful handling of the ritualized physicality when trying to manipulate the process, enhance the expertise, change the communication style, or afford new collaborative practices.

2.2 Physical Interaction: From Rooms of Collaboration to Rooms of Advice

Since the rise of computer-supported cooperative work (CSCW) and group support systems, researcher have struggled to embed the newest technology into physical rooms to improve collaboration. A first generation of researchers embedded desktop computers into specially designed physical furniture to create digital lecture halls [61] and meeting rooms [54]. The rooms with the furniture were not just containers for the technology, but part of a comprehensive collaboration design [57]. As technology became more mobile, researchers moved beyond fixed rooms. Instead, the technology was designed to be adaptable to different needs and collaborative settings, e. g., workshops in different configurations [74] or informal collaboration spaces supported by ‘Roomware’ [80]. At the beginning of the 21st century, it became clear, that computers would not only become ubiquitous, but also should be hidden from the users in order to advance interaction [67]. This insight became the more important, the more intimate and delicate the collaboration is. One of the most intimate and delicate settings is advice giving.

Despite the importance of physical conduct in collaboration, research on the non-verbal conduct in advisory services has focused, primarily, on the table and what is on there [19] as well as the sitting position [35], as depicted in Figure 2 and Figure 3. It lacked a holistic view of the physical matter of an advisory service. However, banks employ building architecture and interior design alike redesign of their leaflets or process changes to transfer the desired picture of themselves as institutions [28]. Rooms dedicated to service provision influence the encounters too. Consequently, designing technology while focusing solely on the material used during advisory services, as it has been done previously [31], [35], [51], can be risky. The studies report on collaboration breakdowns attributed to changes in external conditions such as sitting position [35], [51]. While the changes might well be necessary to suit a specifically designed support system, they distorted the physical order of a ritual. They forced the participants to establish a new ritual. In short, they broke with participants’ expectations or obligations concerning the use of space, such that eye contact or effective conversation were hindered [35], [51].

Figure 1 
              Elements of physical configuration in advisory services.
Figure 1

Elements of physical configuration in advisory services.

The wide lens applied in the current study acknowledges the fact that material, the area accessible to the interactants, and the physical environment surrounding them impact the interaction. On the one hand, they afford specific performances while limiting others; on the other hand, they have been designed with specific routines and rituals in mind. This inherent relation between advisory services and the characteristics of the physical circumstance are related very tightly. The rituals’ occurring depends on the place (rooms, locations, buildings), conditions (light, furniture, sitting positions), and specific elements (documents, sheets of papers, pen) [19], [22], [81]. Figure 1 shows the physical configuration by illustrating the relationships between the physical areas. Note that the particular areas correspond to Kendon’s o-space (interaction area), p-space (access area), and r-space (environment) [46], [47]. However, to differentiate from Kendon, who studied how the spaces and new formations between individuals emerge, we prefer to stick to the more general term area, which also better expresses the static nature of physical configuration in advisory services [7]. In the current study, we treat the physical configuration of advisory services as a source of requirements for a system to be developed.

Figure 2 
              Physical organization of a traditional, pen-and-paper-based financial advisory service [19].
Figure 2

Physical organization of a traditional, pen-and-paper-based financial advisory service [19].

Apart from the spatial and physical configuration of advisory services, we also acknowledge that existing practices can be used as a source of knowledge about the advisory service ritual and provide requirements for a system as well. In particular, when attending to an advisory service, e. g., at a bank, advisees suppose not only a clean and tidy room and an advisor dressed in a specific manner, but they expect that the advisor will take notes, use tools to make calculations, and refer to leaflets and official documents [19]. Even those who never attended an advisory service at a bank rely on an intuition of what the experience may be like [40], [75]. The expectations from one side match the performances of the other side. Many bank advisors (and advisors in other commercial branches) rely on an advisory paradigm called pencil selling. It involves using empty sheets of paper and a pen to illustrate the main message transferred verbally with ad-hoc generated pictures, calculations, or text [25], [76], [77]. The range of particular practices employed in pencil selling is large. Still, they involve writing and drawing on paper, moving and freely positioning paper within the interaction area, as well as removing it from the interaction area and piling it on the side [19]. Overall, the pencil selling method – even though being often popularized as a specific marketing concept – involves traditional means of collaboration and explanation using pen and paper [19]. The pencil selling behaviors have become inherent elements of the current advisory ritual in many sales and advisory services.

Even though pencil selling has dominated advisory services for decades and thus shapes the expectations of many bank clients, this approach receives serious competition. First, after the crisis of 2008, banks are increasingly responsible for educating their clients and documenting this education effort. They want to involve them more in a shared decision-making process [64]. Second, financial products are increasingly complex, such that the drawings and explanations based on brochures cannot capture this complexity [63], [64]. Additionally, many advisors simply cannot draw, such that the pencil selling turns into a calculation exercise: the advisor calculates values in front of a waiting client [19]. Third, face-to-face advisory services receive competition in the form of more powerful online self-services and robo-advice [1].

Figure 3 
              Sitting positions and their relation with phases of an advisory service during collaboration with a table-top display [34].
Figure 3

Sitting positions and their relation with phases of an advisory service during collaboration with a table-top display [34].

If face-to-face advisory services survive in the future, they need to establish a competitive advantage beyond clever sales tactics. We argue that the advantage can emerge as a consequence of an intelligent combination of human and computer, such that the computer complements the human by contributing skills in which computers are strong (e. g., documentation, visualization, calculation) and vice versa [6]. Based on their experience with pencil selling, the advisors are strong in communicating and explaining complex products by using individualized and straightforward information [19]. They are also experts in rapport building [75]. And they established impression management tactics that involve the use of physical material and documents [19]. No computer can take over those tasks easily. However, previous research often disregarded the role of advisors’ skills by creating technology-oriented solutions: they imposed a specific order of actions by representing a step-wise logic of the software [62] or forced the advisor to explicate information about the advisee to enable for further processing [50], [52]. Primarily, however, the prior solutions took the essential tools of an advisor, pen and paper, away from them. The rituals were not possible to conduct, such that advisory services did not unfold in line with the expectations and obligations. This study aims at complementing advisors’ practices – especially the material routines – with IT and affording new performances compatible with the advisors’ embodied work practice. Accordingly, it relies on requirements we derive from the current physical configuration and practice. In the following, we identify those requirements while using Kendon’s [46], [47] perspective on the physical environment to structure them. The requirements are in italics and are assigned a specific number for further reference. We derive the requirements from previous literature on technological support of advisory services and from field observations collected in many years of research on financial advice.

Figure 4 
              A vertical display during a financial advisory service was shown to disturb the natural flow of a conversation.
Figure 4

A vertical display during a financial advisory service was shown to disturb the natural flow of a conversation.

The interaction area in advisory services is the space where participants exchange sheets of paper, collaborate on specific documents and position their materials. Physically, it is simply the part of the table between the advisor and the advisee. Documents and objects that enter the interaction area have a meaning that results from their content and their position in relation to other documents and the participants [19]. Two sheets of paper put parallel to each other are likely to represent alternatives or afford comparison, while a pile of documents lying next to the advisee includes, probably, his documents. This gives the client a chance to understand the ongoing encounter implicitly: they know what to focus on, what pertains to their specific case, and what should stay out of focus. Therefore, it is essential to allow for free positioning and moving of paper and other objects (DR1). At the same time, paper is a medium that keeps its semantics over time. This allows for recognizing previously used documents and easy reference to them [19]. The objects in the interaction area keep a stable, permanent identity. Thus, a solution needs to guarantee object permanence such that content is permanently linked with specific physical objects (DR2). Moreover, advisors rely on a pen to generate individualized content and make notes. This behavior signalizes interest in the client’s specific situation and gives the client a feeling of being treated uniquely [9], [49], [50]. Accordingly, the system needs to accommodate for unique and individualized input, output, and manipulation of data (DR3).

The access area in advisory services is the territory that is accessible to the advisor and the client but is not the focus of the interaction. The access area embraces the table client and advisor are sitting at and possibly some objects in reach like a bag or a folder positioned on a chair standing nearby. The table forms the physical basis for the pen-and-paper interaction: an interaction like described above would not be possible without a table underneath the interaction area. The client and the advisor require the table to distribute their documents across and provide physical support for their position: clients and advisors tend to lean forward towards the interaction space while putting their arms on the table. On the one hand, they put themselves in a comfortable position to focus on the content and the social interaction. On the other hand, they can signalize their interest in the content being presented by taking such a position [34], [51]. To enable those behaviors, a potential solution needs to support physical interaction on, at, and with the table (DR4). Additionally, the table (and, sometimes, surrounding chairs or other nearby furniture) takes the role of storage space for documents and objects which are out of focus at a specific moment. Frequently, the advisors distribute their material all over the table. This way, they do not forget things they wanted to talk about during the advisory service and signalize to the client what content remains to be addressed [19]. They also use the table to store documents they need for the service’s post-processing or files that need to be scanned or copied during the service (e. g., records provided by the client on paper during the conversation). They can stay in sight of the client without entering the interaction space. Accordingly, the developed solution should provide space for storing out-of-focus objects on the table (DR5).

The environment of advisory services embraces the room in which the conversation takes place. Our experience shows that the rooms are mostly very tidy, clean, and modern [19]. They provide space for two to four persons. The table stands in the middle of the room. Sometimes, another piece of furniture (e. g., a chest of drawers) stands at a side with some brochures displayed on it. Usually, there is no or one piece of art hanging on the wall. Technological devices like a printer, scanner, or monitor are located away from the interaction area. A monitor and a keyboard might be positioned at one of the sides of the table, although some banks do not have any monitors in the advisory service room. Other devices might be hidden in the drawer board, below the table, or are not even in the room, such that the advisor leaves the room to copy the documents. The way the rooms are furnished and decorated makes clear that the conversation between the advisor and the client is the core activity that happens inside. An unusual or dominating device like in Figure 4 might disturb this focus and negatively impact the conversation flow [51]. Therefore, a technology employed during financial consultation should interfuse with the physical surrounding (DR6) such that it disappears in the room and does not distract the participants. Yet, advisory service rooms share another characteristic: they are filled with light. Given that pen and paper play an important role during the interaction, bright light is necessary to see what is written on the paper. Consequently, the rooms feature large windows and bright, modern lights. Additionally, in modern branches, banks employ walls made of glass, which makes the rooms even brighter. Accordingly, a potential solution needs to be applicable in bright rooms (DR7).

2.3 Technology in Advisory Services

Overall, the listed requirements make clear that designing a system for use in advisory services is a challenge. Simultaneously, research [16], [18], [36], [62] and politics [63], [64] call for a more effective information transfer and higher transparency in advisory services, such that the advisor and the advisee can engage in collaborative problem-solving. IT can play a major role in this regard. Previous research on supporting financial advice has shown that dynamic manipulation of data can support client education, such that advisees can make more informed decisions and acquire an understanding of the core issues under consideration [35], [36]. It can also contribute towards informational and procedural transparency [62] and help the client share relevant information in a more structured and complete way [49], [52].

Figure 5 
              The use of a tablet during burglary prevention advisory services [11], [20], [30] to feature mobility during an advisory service.
Figure 5

The use of a tablet during burglary prevention advisory services [11], [20], [30] to feature mobility during an advisory service.

Research on the application of IT in other advisory services, such as burglary prevention service (cf. Figure 5), travel planning services, or employment consultations, show further potentials for supporting advisory services. IT offers seamless documentation of the advisory process [30], empowers the advisors to provide more professional [32] and persuasive [11], [20] services, and creates a more engaging experience for the advisee [17], [58]. Overall, there is much evidence for a positive impact of technology during advisory services. Yet, the uptake of modern, multimodal, and interactive technologies in banks remains slow [19], [60]. The likely reason is that the tools, despite their advantages, compromise the quality of interaction between the advisor and the client.

Heinrich et al. [35] point to relationship building as a core value in an advisory service. Accordingly, a goal of an advisor is to establish a lasting and enjoyable interpersonal relationship with the client and vice versa. Simply put, the client and the advisor want to have a pleasant interaction with each other going beyond the fact-based formal interaction. Dolata and Schwabe [19] argue that material practices performed by the advisor aim to impress the client in many ways. For example, they should position the financial advisor as competent, orderly, and customer-oriented yet unique. They argue that technology might compromise any of those aspects and, especially, limit the advisor’s ability to express their unique personality. In a pen-and-paper-based service, they use their individual writing style or even a special pen of their choice for this [19]. With an interface imposed on all advisors, signaling an individual character might even be more demanding. Additionally, technology might introduce additional hassle for the client or the advisor: it might, for instance, force them into ineffective communication and collaboration practices [51], [52] or impose additional cognitive load, e. g., to make sure that the client does not see confidential or inappropriate content [14]. What emerges is a dilemma for the advisors and the responsible organizations: Are the potential improvements worth risking existing routines? Is best the enemy of good?

We claim that part of the problem is that the prototypes and systems employed in research and practice removed the existing practices and introduced a new physical configuration. At once, the focus of the conversations moved away from the interpersonal exchange in which the participants look at each other in a laid-back position to an encounter in which they bow over a (sometimes too small) screen in between them or look at a monitor filled with various information and loose shared focus [34], [51], [60]. Consequently, the real treasure of advisors’ process knowledge, practical experience, and embodied performance patterns seemed thrown overboard. Neither the client nor the advisor could rely on scripts and obligations that follow from the institutional character of the encounter [12], [19], [38]. None of the parties could draw on the collected experience and situational knowledge. In this manuscript, we present a system that preserves the physical, ritualized character of advisory services es expressed by the requirements listed above (DR1–DR7) and tries to complement them with the most natural advantages of a computer: quick calculations, dynamic presentation of data, and easy manipulation thereof.

2.4 Mixed Reality

Mixed reality (MR) responds to combining traditional interaction in a physical space with the potentials of modern computing. MR approaches, like spatial augmented reality [4] or tangible interfaces [43], [44], incorporate physical objects and surfaces into digital interaction to engage spatial perception and allow for touch as a sensory cue. This research promises to combine the material nature of human behavior with the powers of a computer. The relevant efforts flow into a range of mixed-reality discourses: Augmented reality (AR) studies how to overlay physical spaces with computer-generated content [2], [3]; spatial AR projects directly into the environment of the user [4], however, it requires powerful and high-fidelity overhead projectors. Organic user interfaces (OUI) focus on using non-planar spaces that function as means of input and output, such that users can interact with them through bending, folding, and manipulating the form – this includes using paper as a display [41], [42]. Tangible user interfaces (TUI) focus on enabling interaction, manipulation, and collaboration [73] with digital content through physical objects and space [44], [89]. Finally, pen-and-paper user interfaces (PPUI) try to bridge the gap between digital interaction and paper, remaining an essential tool in many human activities [55], [78], [79]. All those areas of research share the interest in extending the physical world with digital content and functionalities. To a certain extent, they want to turn the user’s focus and attention away from the computer and back to the physical world, thus allowing for calm design [84], i. e., one in which the technology provides information or enhances the experience, but does not compete for user’s attention and stays in the periphery until needed. Each discourse offers numerous ideas on how a support system for human-centered financial advisory encounters could look like. However, it remains unclear which suggestions are promising and how to design and develop a combination of them to support the advisory services.

The scientific discourses listed above focus, primarily, on single-user interaction and therefore approach the ease of use from this perspective. However, financial advisory services pose additional problems. Since it is a collaborative situation, the activities of one party need to be accountable to the other party, such that the action can be interpreted appropriately within the context, place, and the applied script or role [22]. Inability to interpret an action will lead to irritations that require attention to resolve [50]. For instance, if an advisor needs to tap three times on his face to launch a specific computation (even if this interaction is easy), the advisee may focus on resolving the meaning of the interaction rather than focusing on the content [22], [23]. With natural and visible data manipulation techniques, the advisee is more likely to understand the impact of specific actions on the financial situation or its representation [35]. Introducing manipulation of figures with a clear meaning along with TUI’s principles seems therefore promising [85]. Furthermore, since the financial advisors are concerned with their impression, any inappropriate or unprofessional action implemented in the system might result in a reluctance to use it. This issue also holds for environmental features: advisory services take place in light rooms at designer tables, such that designing a system for a dark room (as suggested by many spatial AR applications [2], [4]) would break with advisees’ and advisors’ expectations regarding advisory encounters. Additionally, the high-touch nature of advisory services leads to actions and behaviors with fuzzy, implicit meaning – often, the advisor makes notes that the advisee should not notice [11]. PPUI research encounters the same problem: many explicit activities (e. g., inking or drawing a link) have a varying conceptual meaning depending on the context, as well as previous and future activities [79].

Overall, even though the existing literature on MR rarely attends to professional, institutional collaboration, it includes indications for promising interaction mechanisms. However, it remains uncertain how to combine those singular implications into a coherent system for financial advisory services. Specifically, it is necessary to figure out how to overcome typical limitations of state-of-the-art MR systems like the dependency on low-light environment, recognition of relevant and irrelevant objects, or focus on a single-user scenario or collaboration between equal partners outside of the institutional context. Furthermore, the MR interfaces bring along new challenges: form and application of such a system is not a part of a typical advisory encounter script and might thus overwhelm the clients and the advisors who expect something different based on a situational script [12]. Additionally, MR might introduce usability problems leading to rejection of the system altogether. An unreliable touch input during a visit to an amusement park can generate a bit of disappointment and will be forgotten quickly, but an advisor who tries to launch an application for several seconds does not create a picture of a reliable person and the bank. Given the vital relevance of the ritualized character of physicality in advisory services and the importance of impression, there is less room for trial-and-error and less potential for testing early versions of the system in the field.

3 Project Context and Methodology

LivePaper was developed in a collaborative effort between a Swiss regional bank and two Swiss universities to support client-centric advisory services. The project was supported by the Swiss Innovation Agency and aimed at developing a prototype that can be turned into a product after the funding period. This has important implications for the project and the character of the research. First, the technology developed or used in the project needs to be applicable in commercial settings. Second, the development is driven by the needs of the application partner (in this case, it was the regional bank) and its employees, who act as product owners. Third, the ultimate product should bear a promise of commercial success such that it either produces economic returns when used by the application partner or it can be marketed to other institutions. Whereas the bank was providing the necessary financial and practical expertise, as well as contributed advisors who acted as experts during the development and evaluations of the prototype, the universities were developing the system, designing the interaction with the system, conducting evaluations, and transferring the state-of-the-art from research literature to the field. The consortium was working together over a period of approx. 2.5 years, starting with a design exploration phase, followed by two technical design and development iterations, each with an experimental evaluation. Thereafter the system was rolled out to two branches of the bank for a pilot study. This state of the system is described in the current manuscript.

The whole project was inspired by the research through design paradigm [29], [88] and followed the design science research (DSR) methodology [39], [65]. Starting with practical problems of the stakeholders and previous literature on supporting advisory services, solutions were developed creatively and iteratively based on the observations in the field. The project members identified technological potentials and adequately described potential models of advisory services and models of interpersonal interaction and tested them with the practitioners. To inspire creative thinking and identify non-obvious solution paths, methods and approaches from design thinking were appropriated and incorporated in the overall DSR process [15]. The field data collected in the project was analyzed iteratively to identify implicit themes and desires of the stakeholders. Especially data obtained from field observation and intermediate evaluation sessions were approached through various theoretical lenses shedding light on rituals, material practices, institutional roles, and technical requirements towards the system [13]. Combined with the literature, this resulted in the identification of the requirements listed above.

The project used various material representations of the ideas to be developed [8]. In the early phase of the project, scenarios were used as the medium to describe problems and potential solutions [69]. With time, the role of clickable and functional prototypes increased, and intermediate testing with key users was used to inform the following steps, according to the constructive design research [53]. Scenario-based and constructive design research have been widely used in HCI for years [87] and belong to its methodological inventory. This guarantees the acceptance of the solution by the stakeholders.

The technical challenges and potentials were explored in parallel to the user-centered activities. One senior member of the project, supported by two other persons, oversaw identifying alternative technical solutions to meet the stakeholders’ needs. Each potential solution went through a double review process. For instance, to determine the appropriate pen input (see Section 4.3 below), the technical and user-centered assessment was conducted independently. On the one hand, the technical abilities and implications were tested with test scenarios created based on the input collected in fieldwork and additional extreme cases. In the case of pen input, this provided information about the latency or precision of the optical character recognition and technical interfaces and their compatibility with the rest of the system. On the other hand, the stakeholders (especially advisors) were confronted with technological alternatives to collect their feedback. In the case of pen input, this involved presenting various combinations of input devices (pens, active boards, paper types, etc.) to the advisors, gathering their reactions, and analyzing them. Where it was not possible to confront the stakeholders with the actual products or prototypes (e. g., it was not possible to present them with all potentially useful overhead projectors), they were confronted with various forms of simulation (e. g., multiple levels of brightness and resolution). The final decision on what solution to choose to respond to a challenge was made in workshops involving senior representatives of all project partners. In the following, we provide an overview of the system as developed while following the procedures described above.

4 Challenges and Solutions

The main objective of the developed system was to support mortgage advisory services. A person who considers buying a property but lacks the financial means to do so can apply for a mortgage loan offered by commercial banks. Usually, mortgages have lower interest rates than other retail credit, thanks to the low-risk profile for the bank. If the client cannot pay back the loan, the bank can monetize the property to cover the outstanding debts. Yet, a mortgage is available only to some clients who fulfill specific financial requirements: in Switzerland, they need to provide funds for 20 % of the property, the monthly rates cannot surpass a particular percentage of their monthly income, they have to assure further guarantees (e. g., access to money from their retirement fund), and there might be additional limitations concerning, e. g., their age or form of employment. Furthermore, there are various forms of mortgages differing in interest rates (fixed or variable), length of the mortgage contract, or ratio of repayment. And those forms can be mixed with each other. This makes selecting an appropriate offering for a client particularly challenging, and the complexity is growing. Thus, LivePaper should support the advisor and the client at determining the client’s debt sustainability to make sure that they can apply for a mortgage at all and help them creating the right mix of mortgage solutions in accordance with the client’s needs. Additionally, it should support the advisor in collecting relevant information about the client and the property to be covered with the mortgage. The design requirements listed earlier have a direct impact on the implementation of the system. Figure 13 presents LivePaper in use during an advisory service, while Figure 6 depicts the technical configuration of the system. This section discusses the technical challenges resulting from the design principles and desired functionalities, as well as our solutions for addressing these challenges.

Figure 6 
            Technical constituents of LivePaper and connections between them [9].
Figure 6

Technical constituents of LivePaper and connections between them [9].

4.1 Environment: Room and Furniture

DR6 demands the system to interfuse with the physical environment. Accordingly, changes to the environment had to be as small as possible. Together with an interior architect, a ceiling-mounted unit (see Figure 7) was designed, which integrates a 4k projector (Optoma UHD60 (Optoma)), an MS Azure Kinect sensor, both pointing downwards, as well as the computer running the software. The bottom surface of the unit can be lit by LEDs, thus serving as a lamp and further concealing its purpose. The whole unit can be moved down and up to easy access to the specific devices. This solution was preferred to full integration in the ceiling for two reasons: (1) the sensors and the overhead projector work best if they are in a specific distance range to the projection surface (table), and regulating this distance might be necessary, (2) this construction can be easily replicated in other rooms or other branches of the bank without essential changes to the room itself. Yet, some cover elements (e. g., the color of the frame) can be adjusted to fit the room and make the impression that the unit belongs to the room.

Figure 7 
              LivePaper unit lowered from the ceiling during maintenance.
Figure 7

LivePaper unit lowered from the ceiling during maintenance.

A flat surface underneath the unit, usually a table, acts as the screen for the projected image. While a homogenous, light gray, or white-colored table surface will increase the contrast of the projected image, the system is designed so that it also works on darker or textured surfaces. This, again, should support the applicability of the system across various physical settings. For instance, a wooden table is the projection surface in one of our three system installations. Projecting on darker surfaces works equally well because the essential information will be limited to sheets of white paper which can be arranged freely on the table. This solution was preferred to projecting directly on the table for several reasons: (1) interacting with paper feels more natural rather than interacting with projected content, (2) paper provides a constantly white background such that the projections are easier to see and read, (3) it is possible to write on the paper. Only auxiliary user interface elements are projected outside the paper boundary onto the table surface directly. Overall, the above choices respond to the DR4 such that the physical interaction can happen on, at, and with the table. Even moving or rotating the table, changing its size, or moving to a different sitting position is possible without any losses if the table is large enough to accommodate the projection. Each time the system starts, it conducts an automatic calibration sequence to adapt to smaller changes in the distance between the table and the projector and sensors.

Another challenge is maintaining contrast between the projected image and the reflected ambient light in a bright room. This is mitigated twofold: firstly, the 3000-lumen output of the projector is concentrated on a relatively small area (approximately 1.5 m2). Secondly, the visual design uses high-contrast colors and white-on-black text. This generates the impression that bright and crisp graphics or letters are presented on light-grey paper. This accommodates for DR6 such that the system can be effectively employed in bright rooms. The design team has tested various color schema, fonts, and sizes throughout the project and arrived at a combination of various light-green, light-blue, white, and yellow shadows as the optimal combination, which is well visible but at the same time not too aggressive. Blue, green, and white elements are used for feeling areas or text, while yellow is used for highlighting.

Figure 8 
              LivePaper system with three active papers, pen input, and tangibles.
Figure 8

LivePaper system with three active papers, pen input, and tangibles.

4.2 Access Area: Paper Tracking and Movement

Since paper is one of the critical elements in this advisory setting, specially marked paper, so-called active paper, is used for interacting with dynamic digital content. The system recognizes it by a narrow bar code on its short edge, which is also used for distinguishing between active paper and other paper documents that may be part of the advisory setting, such as brochures or financial records. Active paper is a consumable that should retain all the properties of an ordinary sheet of paper, such as drawing or writing. Therefore, we provide PDF documents with barcodes that can be printed with conventional office laser printers for an inexpensive way to produce them. This solution was favored over the special paper to support paper rituals of an advisory service, which relies on an unlimited supply of paper.

Several approaches for active paper application development ranging from simply exposing an API to implementing a fat client have been evaluated. Finally, we settled for a Chromium Embedded Framework (CEF) based solution. CEF allows rendering web pages offscreen with the same technology as Google’s Chrome browser. By linking the URL of a web page to the bar code of an active paper, the digital content gets attached to a physical object. The system tracks active paper, and the web page’s content is rotated and translated according to the orientation and position of the corresponding active paper. This contributes towards free positioning and moving of paper (DR1) while at the same time assuring that the content is permanently linked with specific physical objects (DR2), in this case, a sheet of paper. Furthermore, it accommodates for storing out-of-focus objects on the table (DR5). Even if a sheet of paper is moved away from the interaction space and held somewhere on the table, its bar code will be recognized whenever it re-enters the space, and the belonging content will be presented.

In this context, tracking the movement and rotation of the paper turned out particularly challenging. Finding the borders of a white paper on a potentially white table surface is almost impossible due to the low dynamic range of the camera system. Therefore, we rely entirely on detecting the printed bar-code, which also encodes the orientation of the paper. The Kinect sensor features an HD RGB camera as well as an IR camera. While the RGB camera offers sufficient resolution to detect and decode the barcode, we observed that the high exposure time of the RGB camera blurs all objects in motion, making object following impractical. The IR camera has a much shorter exposure time and produces sharp images at a low resolution, such that only detection but not decoding of the barcode is possible. Consequently, we use the IR camera to detect potential barcode candidates, which then, in a second step, are verified and decoded using the high-resolution RGB camera as soon as the paper becomes stationary. We maintain a probabilistic model of the papers on the table while papers are in motion. This model gets updated and is corrected as necessary when papers are positively identified through the RGB camera. Redundancy in the barcode and the depth image of the Kinect sensor are used to deal with partial occlusions of the barcode, e. g., by hands reaching over the paper or when papers overlap.

In addition to the spatial limitations of the camera, we also had to address the inherent latency of the system. The infrared camera runs at 30 frames per second which introduces 33 ms of latency in the worst case. Our internal image processing can take up to 50 ms and the projector, independent of its settings, adds another 64 ms until the image is displayed. This results in an overall latency of approximately 150 ms. During our latency measurements through an independent high-speed camera (FL3-U3-13Y3M-C, Point Grey, 2016) observing the table, we were surprised by the speed at which users moved objects around. Velocities up to 1 m/sec were perceived during these tests. Therefore, the projected image was trailing several centimeters behind the physical object in motion due to the overall system latency. Three approaches were evaluated by a qualitative study (N = 9) to address the trailing image:

(1) Motion prediction: using a Kalman filter [33], motion data is used to forecast object position. While this technique works well for an object in constant motion, it tends to overshoot during deacceleration. The effect is similar to inertial scrolling found in today’s touch-based interfaces. Only two of the subjects liked the playful bounce effect of that implementation but still favored the other two approaches.

(2) Motion blurring: motion blur is the result of the limited temporal resolution of the human visual system [83] or the exposure time of a camera system. Details of objects in motion cannot be resolved as accurately as for static objects. Hence, we had the idea to artificially blur the projected image if the corresponding physical object is in motion to simulate the effect. The problem with this approach is that motion blur looks unnatural if the user is actively following the object in motion. In this case, everything else appears blurred due to physiological limitations. The object in motion, which should be sharp because it is followed by the user, is artificially blurred by the system resulting in visual inconsistency for the user. Only one subject preferred motion blur over the third approach.

(3) Fading: similar to motion blur, the visual appearance of the projected content is artificially altered while in motion. We adjust the transparency of the content by the velocity of the corresponding physical object. Static content is projected opaque while fast-moving content becomes fully transparent (fades out). Despite the fact, there is no real-world analogy of objects disappearing while in motion, eight subjects out of 9 preferred this implementation which in turn was selected for the final implementation.

The design decisions concerning the tracking and visualizing the movement of active paper across the table contribute towards enabling physical interaction on, at, and with the table (DR4).

Figure 9 
              Pen input with abbreviations.
Figure 9

Pen input with abbreviations.

4.3 Interaction Area: Pen Input and Tokens

Notably, the active paper should not only be a presentation surface but also allow for unique and individualized input, output, and manipulation of data (DR3). There are various aspects related to this requirement: providing new information (e. g., the name of the advisee), manipulating existing data including a choice of a data point to be changed (e. g., choosing and changing monthly interest rate), and presenting individual output (e. g., signalizing a recognized name). This all should be possible while accommodating the fact that the paper and other objects can be freely positioned and moved around the table (DR1) and keeping the link between specific content and a physical object permanent (DR2). In the following, we reflect on the various input and manipulation mechanisms featured in LivePaper.

In the spirit of supporting existing pen and paper techniques, handwriting input to the system was mandatory. The limited resolution of the camera system excluded an image-based approach. We use a Wacom Bamboo slate together with the Microsoft handwriting recognition engine (Microsoft) instead. This was a preferred decision compared to structured input or keyboard. During the contextual inquiry, it became apparent that paper forms were rarely used and would have an excessively structuring impact on the dialog with the customer. The keyboard was considered a foreign body on the otherwise clean and non-technological surface of the table. We also discovered that some advisors were using abbreviations for tagging specific numbers (e. g., annual income, house price, etc.) as a memory aid. Based on these observations, we adopted the practice and introduced a standard set of abbreviations for data input (e. g., “STR” for street, see Figure 9). This allows unstructured data input while still attaching the semantics to the data required for further processing by the system. The Wacom Bamboo Slate, which generates an electromagnetic field to record movements of a pen on paper, was preferred over solutions using dedicated paper and a sensor or camera in pen (e. g., Neo Smartpen or Livescribe pen) as well as IR-based solutions (e. g., Fujitsu Mobile NoteTaker). Whereas the Wacom Bamboo Slate (and similar solutions) can be hidden behind any paper block and deals as a pad, the other solutions are either visible such that a clip is added to the top side of the block (IR-based solutions) or require the user to write on specifically prepared paper (including a specific pattern). The latter would make it difficult to write on brochures or use the paper with the bank logo, while the former would again introduce a clearly visible and unusual technical artifact in the interaction space.

One of the key benefits of dynamic digital content over static printed content is its interactivity. Apart from adding new data, it might be necessary to interactively manipulate the projected data on the table. This might be data coming from the core banking system (e. g., the current interest rates) or defaults that require adjustments to the client preferences (e. g., the duration of a mortgage). However, before the user can even manipulate the data, they need to choose what item should be adjusted. Based on a solution proposed by Xiao et al. [86], we implemented finger touch detection. Touch events on an active paper are translated to mouse events for the corresponding web page. Even though pressing a button on paper is not a natural form of interaction, earlier research shows that touching and using deictic gestures with paper occurs naturally within an advisory service. We decided against using a dedicated pointing device (e. g., a special pen or wooden stick) to reduce the cognitive overload of the user: point and click with a finger was considered least obstructive.

Figure 10 
              Two tokens and a an active paper sheet.
Figure 10

Two tokens and a an active paper sheet.

Figure 11 
              The house with the map.
Figure 11

The house with the map.

Having selected the specific item, the user can manipulate the data using tokens. We introduced several tokens with different functions. Specifically, we use a Microsoft Surface Dial for changing numerical values at various places in the application. The Dial is tracked by the IR camera together with the Kinect depth sensor. It is associated with an active paper by its proximity to the paper and a specific numerical item through the touch input. Turning the Dial results in mouse wheel events for the corresponding web page, which can use the events for animating content. The turning action in the physical environment, together with the animated content, helps to understand and interactively explore the underlying financial models.

Additional tangibles include a stylized house roughly the size of an espresso cup and several flat tokens representing different types of mortgages. The house serves as a tangible link for discussing the property and its surroundings. It is tracked by an infrared marker on the top together with the depth image of the Kinect 2 sensor. Similar to active paper, the house can be linked to an URL. Usually, we show a map centered around the property’s location in discussion (see Figure 11). With the same two-step process used for the LivePaper barcodes, tokens are tracked by the IR camera, and three different token types are identified through the RGB camera. Instead of barcodes, we used engraved text (e. g., “FIX” for a fixed-rate mortgage) for token identification.

Together with the tokens, we also introduced a novel interaction technique called “place and link.” In contrast to the static link between an URL and the barcode of an active paper, newly placed tokens on the surface can be linked interactively by a touch gesture to digital content. This link is then visualized by the system as a curved line (see Figure 10). The concept of “place and link” underlines the logical connection between two elements with interaction in the physical environment and is thus easier to comprehend for an observer than spontaneously appearing interface elements. Overall, the introduction of the tokens and actions associated with them provides means for manipulating the presentation’s content and enhances the overall feeling of interactivity. The actions associated with the tokens go beyond the traditional pen-and-paper interaction and, therefore, a new, physical form was chosen to make the intention clear to the user.

Figure 12 
              Using the Dial to change a selected value.
Figure 12

Using the Dial to change a selected value.

Figure 13 
              LivePaper used in an experimental advisory service.
Figure 13

LivePaper used in an experimental advisory service.

4.4 Evaluation and Use

The system described above was evaluated in two design experiments (cf. Figure 13) and subsequently used in two branches of the partner bank. The experiments [9], [10] provided evidence that the system, in combination with appropriate practices, has the potential to significantly improve various aspects of financial advisory service. For instance, advisory services conducted with LivePaper outperform traditional services in terms of client’s satisfaction with the service, their assessment of the information transparency and interpersonal interaction, and the overall perception of bank service quality [10]. For detailed analyses, we refer the reader to the previously published articles [9], [10], [21]. Overall, the clients and the advisors preferred the LivePaper-supported advisory services over the traditional ones.

After the successful experimental evaluation, two bank branches were equipped with the system to be used in mortgage advisory services for four weeks. Two advisors were trained to use the prototypical system outside of an experimental setting. After four weeks, we conducted interviews with the advisors. On the one hand, one advisor used the system only sporadically, with few clients only, and explained that her reluctance to use the system was related to her insecurity and technical problems with starting up the system, as well as lack of “appropriate clients” who would understand that the system has some limited capabilities. Indeed, at that time, the system was not fully functional because it lacked integration in the core banking system – the simulations and orders made through LivePaper would need to be manually transferred to the banking system, and the organizational processes for doing so were not clear yet.

On the other hand, the second advisor developed long-term practices for using LivePaper and even trained his trainee to use the system regularly. He was reporting that “not every client is right for being advised with LivePaper” and “the understanding for who will enjoy it and who will be overwhelmed by it is important.” Still, he was assessing to use it roughly in 40 % of advisory sessions. He admitted that working with LivePaper still requires a significant time for starting up the system, but it became part of his routine: “In any case, you would want to see before an advisory session if the room is clean and tidy. So, I just went there a few minutes earlier and started the system. I can use the time to think about how I want to interact with the client. I launch the system and then position the house token in the middle. It makes a great impression when the client enters the room, and their house with the map is already on the table. I think it looks great, and clients are simply ‘wow’.” He explained that he uses the system primarily to explain the basics of the mortgage system in Switzerland, which he was previously explaining on paper while using hand-made drawings. Later, when the conversation moves towards a specific mortgage offer, he would turn off the LivePaper no to mislead the client by some invalid or out-of-date numbers or calculation models that were hardcoded in the system.

Apart from using the system for advisory services, the bank’s innovation department and management were frequently presenting the system to existing and potential partners. Given that the bank is active in a small region and possesses only 12 branches serving an area with about 500’000 inhabitants, it was looking for additional partners who would participate in further development of LivePaper and share the costs. Meanwhile, it partnered with another regional bank with more than 50 branches serving an area with more than 1’000’000 inhabitants. About one year after the pilot study with two branches, the banks decided to move the project towards a product. By now, in 2021, they have established a startup, which developed a second version of the system to address some technical issues (like refresh rate), update the technological standards, interface the system with core banking software, and add further functionalities. They use the second version of the system regularly in more than five branches and are successful at attracting interest from additional partners from Switzerland and beyond. Yet, the design and technical solutions presented in this chapter remain essentially unchanged.

5 Discussion

The case of LivePaper shows how much work and technical know-how is necessary to create an immersive and pervasive experience that uses rather than replacing existing physical rituals and practices. The main challenge in augmenting natural experience with technology is human beings’ incredible mental and physical ability. It is very difficult to establish an impression of immersion between physical objects and digital content, as exemplified by the challenge created by the inherent latency of the system described above. This suggests that more research in MR is necessary, and even more evidence is needed as to what technical solutions create the intended impression for the user. Yet, the complexity grows significantly if we add the social and institutional components into the equation. A system like presented here does not possess the ability to infer the status of social interaction or institutional scripts used at a given moment in time. It is incapable of situated adaptation. Nevertheless, it impacts those scripts and the ongoing interaction. The solutions presented here offer one way to deal with those difficulties: the system remains very passive, almost invisible throughout the interaction until purposefully included in the interaction. This happens by moving a piece of specially marked paper or a token into the interaction area. If this is not the case, the system that runs in the background can only take on the input advisors make on their pad by pen input, but an unaware client will take this as a natural note-taking practice.

Due to the non-invasive character of the system, the advisors establish new practices for introducing the system [21] and fading it out. This implies that, on top of managing the interaction with the client, they need to actively manage the interaction with the system. They need to think to decide which client the system will fit, how and when it should be introduced, and whether it has limitations that would make using it for some parts of the service somewhat cumbersome. This, combined with technical problems, might reduce motivation to use the system regularly and explains some of the issues experienced by the other advisor who had the chance to use the system in the actual setting. Yet, banks and other financial institutions’ interest in using the product also shows how much interest there is to improve the advisory services and make them more appealing and efficient. This has implications that go beyond the design and development of the system.

In the following, we attend to various implications of the LivePaper project. After that we propose a concept of an agentic system that features situated awareness and discuss its potential to further improve the advisory service.

5.1 Implications for Advisory Services

The research confirms the important role of physical rituals in financial advisory services. As opposed to some of the earlier systems [34], [50], [51], [52], [62], LivePaper was considered a natural yet fascinating supplement to a service rather than a disturbance which changes the character of the service. The research concerning the selection of features and technical solutions shows that stakeholders preferred a system that merges not only with the physical environment but also with the practices that happen in this physical setting. This confirms that people follow scripts about how institutional encounters should look like and establish expectations based on this [12], [19], [37], [38], [81]. The development of new practices which might lead to new shared rituals requires time and a gradual approach. This might be exemplified by the new approach developed by one of the advisors: making a strong point by using the system at the beginning but then returning to the more traditional, paper-based approach. We claim that many of the solutions developed for use in advisory services ignore the subtle yet important role of physical and institutional rituals and focus more on supporting procedures and processes of an (optimal) advisory service. They might provide the right functionalities, yet they will feel quite foreign and detached. They are redefining the whole encounter at once, without making space for existing practice and thus the feeling of continuity between what is old and expected and what is new and inspiring. This research shows that advisory services are very settled and ritualized encounters. Changing them will not happen through formalizing or enforcing any regulatory or technical requirements proposed in the literature or law [63], [64] but requires a longitudinal approach. Technology bears the potential to support this evolution to a certain extent by providing means to fathom what is acceptable for clients and advisors and what overwhelms them.

Whereas technology can support establishing new practices and incorporating them into the existing rituals, the diffusion of those practices requires effective training. Advisors are frequently lonely rangers, who learn about the practice of consulting clients from more experienced peers at the beginning of their career, but later only rarely receive uptraining and yet more seldom they receive feedback to their work as an advisor [11], [20]. This limits the natural propagation of effective and identification of ineffective advisory practices: each advisor establishes their unique practices and narratives. The introduction of a novel technology is one of the very few interventions which provide an opportunity to change the status quo. Yet, frequently, advisors are left alone with the system and develop not only different ways of dealing with the technology but also radically different attitudes, as presented above. We claim that if advisors were given a chance to learn from each other on how to use the system and share their experiences for real advisory sessions, rather than artificial training situations or experimental sessions, the diffusion of technology and of effective practices would be more successful. We thus call for a practice-oriented, iterative, and professionally facilitated sharing of expertise and experiences between advisors, especially when novel technology is being rolled out. A training session like this could involve the following: (1) discussion of positive and negative episodes from the encounters supported with the system, (2) identification of exceptionally effective and ineffective behavioral patterns based on those episodes, (3) practice sessions in which the advisors provide advisory services to each other in which they enact the identified practices such that they can experience how those practices “feel” from the perspective of the advisor and from the perspective of the advisee, (4) reflection about the practice sessions. Indeed, we implemented a similar approach towards training when preparing advisors for experiments with the difference that we provided them with imaginary scenarios about episodes rather than actual episodes. The advisors liked the training, and their effects were observed in the advisory services they provided during experiments, but a systematic evaluation of this training methodology is yet to come.

5.2 Implications for Technology in Advisory Services

Whereas some of the challenges experienced during the development of LivePaper result from the overall architecture and the decision to create an MR solution, many of them relate to the nature of advisory services. The latter challenges and implemented solutions are transferable to other systems for advisory services support, even those that rely on a different technological stack and use different interaction mechanisms. First, LivePaper combines two aspects of advisory services: marketing [19], [49] with information transfer [35], [62]. Marketing is addressed by elements helping with impression management, such as the token of the house and the map and personalized and visually appealing visualizations. Information transfer is supported by dynamic manipulation of graphics so the client can understand the implication of her decision as well as meaningful, easy-to-understand, and stable semantics of the tokens and papers on the table. The research on supporting advisory services has mostly treated those two aspects in separation, thus yielding systems that helped transfer information and educating the client, but were lacking improvements in marketing aspects and vice versa [10], [36]. LivePaper overcomes this divide and can improve marketing-relevant measures and pragmatic measures of a service encounter [9], [10]. This implies that technology for advisory services needs to acknowledge the fact that financial advisory service is simultaneously a selling encounter and education or problem-solving encounter [18]. Balancing the two is a challenge itself because selling and marketing frequently rely on information asymmetry, while information transfer tries to overcome this asymmetry. LivePaper thus deliberately focuses on making the right impression and inciting joy sparked by learning new things in an interactive manner. We are convinced that this approach is a promising and ethically valid way to improve advisory services not only in terms of their quality but also in terms of their economic value and, ultimately, sales.

Second, LivePaper positions the relationship and interaction between the advisor and the client at the core of the service. On the one hand, this results from hiding the technology and making it available and visible only when it is needed and actively used. On the other hand, the system inspires discussions by providing information previously unavailable in a service, like a map showing the location of the property and its surroundings. Consequently, discussions about the property under consideration emerge [9]. This gives the advisor a chance to show their interest in what is important to the client, and the client can express her motivations. We claim that technology should provide serendipitous information and so spark exchange between the advisor and the client. This might be achieved with as easy means as a map. Still, more advanced interventions are possible, where the content gets displayed or suggested based on the information about the conversation available to the system (e. g., the client’s address). This might function as an icebreaker and help the participants establish a relationship that relies on mutual recognition. The past literature already acknowledged the importance of a free conversation phase at the beginning of an advisory service. Specifically, it was shown that this phase is important to create an offering that fits the client’s implicit needs and values [49], [63]. We claim that role of the technology goes beyond storing and processing the collected information, but it might also inspire the discussion. Yet, the border between inducing a free conversation and forcing participants into ineffective and unpleasant practices (form filling) is very thin [52]. Thus, more research is needed on how free conversations can be effectively supported without creating a feeling of enforcement or coercion. LivePaper’s house token can inspire exploring other approaches.

LifePaper also advances traditional hardware supporting collaboration in rooms [61], [67], [74], [80]: Technology is hidden not only to the point, where only the interface is visible, but it completely disappears and only an augmented natural environment remains. Thus, it enables to maintain the delicate and intimate relationship between advisor and client and at the same time makes modern tools and information accessible during advisory service encounters. Such approaches could not only improve financial advice, but may also be applicable in settings like travel advice [59] or medical advice [26].

5.3 Implications for Mixed Reality

Finally, the design of the system and choices made along the way can inform the development of MR solutions and research related to them. It provides specific ideas on how to solve practical problems related to creating an immersive yet professional experience. First, it presents how the devices can be disguised to make them disappear for the user. Second, it suggests an approach to move objects and sheets of paper around the table, so they feel and seem natural and synchronized, despite latency. Third, it recommends specific colors to be used when projecting on paper and table surfaces, such that projection works and looks appealing on white and wood surfaces. Fourth, it provides an example of how PPUI can be combined with AR to create a coherent paper-based yet interactive and dynamic experience for the user. Fifth, it allows for multimodal yet unobtrusive input mechanisms. Sixth, it indicates how content and physical objects can be linked to each other, such that the links remain stable even if the physical objects are moved out of interaction space. Seventh, it establishes an overall framework that allows for dynamic input to projected content without making boundaries of the interaction area explicit. Based on the design of the system, we can identify the following design recommendations:

  1. Use usual elements that people expect in the environment, like a lamp for hiding the technology.

  2. Combine motion blurring with fading to create the impression of a synchronous move of a physical object like paper and the projection linked with this object.

  3. Employ white and other bright colors for small elements and text. Use darker colors for filling areas and plains.

  4. Provide simple sheets of paper as input and output surface yet make active sheets recognizable to the system by using individual bar codes.

  5. Use pen input with optical character recognition for entering specific and precise information; use tokens for manipulating projected content in a dynamic way.

  6. Associate each active sheet or active token with an URL that links to the website featuring the content connected to this item.

  7. Allow each website linked with an item to appear on a black plane without visualizing the borders of this plane.

Whereas designers and developers surely find further inspiration in the description of the system above, we claim that those are the core points behind the technical design. They embrace various levels of design stack: the physical devices (1), the software architecture (6, 7), as well as the interface design (2, 3, 4, 5). We admit that previous examples of MR systems were implementing some of the above recommendations already. For instance, PPUI research has been using paper for input and output for a long time [55], [78], [79]. However, in many cases, it assumed all sheets of paper to be an active system element. To guarantee that advisors can use documents provided by the client or bank’s brochures without triggering a projection, LivePaper introduces bar codes printed at the edge of normal paper sheets to make them traceable and recognizable. Similarly, OUI [41], [42] and AR [2], [3], [4] systems already used the trick with a projection of a black background plane to create an impression that the projected content items are projected separately rather than as parts of a single large picture. However, LivePaper extends this illusion such that the content appears permanently linked to individual sheets of paper. Finally, TUI [44], [89] has a long tradition of using tangibles and combining them with projection to allow for embodied interaction. Still, LivePaper differentiates between precise input via pen and approximate manipulation of projections with tokens, including the Dial. Overall, we claim that the novelty of LivePaper does not emerge from the individual atomic design decisions but more from a coherent combination of the available features. Importantly, LivePaper uses off-the-shelf technologies such as Microsoft Kinect or the Bamboo Slate, which implies that MR is within reach for developers and designers. They do not need to tinker with self-made devices to create an immersive impression for the user. Yet, combining and synchronizing between the many input and output modalities keeps being a challenge and requires clever approaches and programming skills.

5.4 Digital Agents in Advisory Services

Over the last decade, the research on technology for advisory services points in one direction: less is more. Specifically, the solutions developed and evaluated over that time moved from clearly visible and dominating devices shown to impact the basic structure and rhythm of the conversation [16], [18], [52], [62], [75] to technologies which disappear and align with pre-existing practices and rituals while extending them only where necessary [9], [10], [21], [60]. This shift aligns to a certain extent with overall technological trends. Yet, it also is a symptom of accumulation of knowledge and evidence suggesting that invasive technology takes too much space while destroying the very subtle construction of institutional talk. We envision two alternative paths for the future development of this research area.

As argued before, LivePaper moves much responsibility to the advisor and provides very limited guidance. Advisors must sense the client and decide when and how they use the system or keep it invisible. According to the collected insights, three factors are key: (1) is the client at all open for technology in advisory services, (2) does the current stage of the conversation allow for meaningful use of technology, and (3) what functionality of the system should be used and how. One can imagine that decisions concerning (2) and (3) can be effectively supported with situated contextual awareness. Based on multimodal information about the ongoing conversation (voice pitch, transcription, keywords, gesticulation, exchanged documents, inputs made), the system could estimate the phase of an advisory encounter or derive what content is currently salient. It could then use the information to become more active, i. e., project relevant data or create notes in the background or disappear. This would reduce the cognitive overload on the advisor’s side, who could rely on guidance from the system. Thanks to the progress in machine learning and ambient intelligence, this idea could become a reality soon. It would make the technology disappear even more – not only for the client but also for the advisor – yet accessible when advantageous. This would keep many of the social and physical rituals intact and augment them in the traditional vision of institutional talk.

However, if we were able to make the technology smart enough to sense when it can offer a reasonable contribution, why not question the trends of hiding the technology explained before. One could assign conversational agency and institutional identity to the technology. In other words, the system would become a third partner equipped with specific abilities like quick calculation, note-taking, or overseeing the legal aspects of a specific transaction. Rather than disappearing and immersing with the background, it could possibly have a physical or visual representation and engage in conversation with the other two participants. While surely controversial, this vision finds support with some advisors [12]. Importantly, including a third party with its own identity and responsibilities, different from that of an advisor or a client, would require a deep rethinking of rituals and practices typical for an advisory service. Currently, the service encounters follow the ideal of institutional talk, such that appropriate and expected behaviors and contributions follow the institutional identities of the participants: the advisor acts as a selling person and a representative of the bank, while the client acts on behalf of her own and, sometimes, her family and seeks a product offering or a bit of adequate advice. The scripts of institutional talk do not foresee a third, independent party with very specific yet limited abilities. This bears challenges beyond visual and interaction design: What is the appropriate conversational behavior of a digital agent in an institutional conversation? What is its institutional identity? How can the identity be signalized to the other participants? How to make the behaviors consistent with an identity? How to engineer the collaboration between human and non-human participants? How to guarantee a natural flow of conversation? Given the rapid development of natural language processing and digital assistants, those questions require an answer and point towards new challenges in supporting advisory services.

6 Conclusion

Supporting advisory services is a challenging yet still underrepresented area of research. Yet, the more technology will enter such encounters as patient-doctor consultation [26], crime prevention [20], or public services [71], the more this line of research will catch the attention of researchers and practitioners alike. This article exemplifies technical and design challenges that emerge during the development of a system for financial advisory services which aims to support rather than replace existing physical practices and rituals. The above guides developers and designers of such systems while calling for more attention to details of the interaction between the parties and their physical surroundings. Similarly, it provides valuable technical insight and inspiration for developers of MR solutions. It suggests some novel configurations to respond to known problems, including latency management or linking between physical objects and digital content. Additionally, the article can inspire further research on advisory services and ways to effectively support them. Specifically, we call for a more profound consideration of possible futures and how researchers can help to shape them. Despite its emerging commercial success and positive evaluation, LivePaper leaves much space for further development and conceptual and technical improvement. We invite the community to take on this idea and move it beyond the current boundaries.

Award Identifier / Grant number: 17716.1 PFES-ES

Funding statement: The research described in the current article was supported by Innosuisse – Swiss Innovation Agency (project 17716.1 PFES-ES).

About the authors

Mateusz Dolata

Dr. Mateusz Dolata, born 1988, is a full-time postdoctoral researcher at the University of Zurich, Department of Informatics. His research interests span co-located collaboration in IT-supported settings and application of artificial intelligence for the common good. He applies a multidisciplinary perspective shaped by his background in computational linguistics, philosophy, and applied computer science. He has co-authored numerous conference and journal articles which open the black box of human work practices in human-computer assemblages. His research has appeared in journals and proceeding series including Computer Supported Cooperative Work, Proceedings of the ACM on Human-Computer Interaction, Information Technology & People, or Business & Information Systems Engineering.

Simon Schubiger

Dr. Simon Schubiger works as a senior principal software engineer on 3D technology at Esri. Before, he was professor for computer graphics, teaching game design and audio/video processing at the University of Applied Sciences Northwestern Switzerland and was the head of software development at Esri R&D center in Zurich, Switzerland. He is co-founder of the ETH spin-off company Procedural Inc, acquired 2011 by Esri. Previously, he was lecturing mobile system architectures at ETH Zurich, worked for Swisscom Innovations, and as an associate researcher in the Pervasive and Artificial Intelligence group at the University of Fribourg. His research interests include 3D computer graphics, multimedia performance systems, mobile computing, knowledge representation, programming languages, and user interface design.

Doris Agotai

Dr. Doris Agotai has headed the newly founded Institute for Interactive Technologies FHNW since 2018, which is a multidisciplinary informatics institute specializing in the development of digital interfaces for people and processes. Until the end of 2017, she was deputy head of the Institute for 4D Technologies and, as the person in charge of the Design & Technology research area, built up and established the focus on design in computer science. In this role, she realized numerous projects in applied research and development and, in addition to regular teaching engagements, accompanied student projects, including the interdisciplinary and international iPOLE project. Her research interests embrace design and deployment of bleeding-edge technologies in collaborative settings.

Gerhard Schwabe

Dr. Gerhard Schwabe has been a full professor at the University of Zurich since 2002. He has studied collaboration at the granularity of dyads, small teams, large teams, organizations, communities and social networks. In doing so, he follows either an engineering approach (“design science”) or an exploratory approach – frequently in collaboration with companies and public organizations. He has published in computer science conferences as well as in major information systems conferences and journals. Currently, his research interests focus on blockchain applications and human-robot collaboration.


We wanted to express our gratitude to all collaborators who participated in this project. Our special thanks go to Ulrike Schock, Fiona Nüesch, and Dr. Mehmet Kilic who made significant contributions to the design of LivePaper. We also want to express our gratitude to the Hypothekarbank Lenzburg for sharing their expertise and supporting the evaluation of the system. We thank the editorial board of i-com for providing a platform to publish this work.


[1] Arwas, A., & Soleil, K. (2016). Robo-Advice 2.0: The Next Generation. Journal of Financial Transformation, 43, 30–36.Search in Google Scholar

[2] Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6(4), 355–385.10.1162/pres.1997.6.4.355Search in Google Scholar

[3] Billinghurst, M., Clark, A., & Lee, G. (2015). A survey of augmented reality. Foundations and Trends® in Human–Computer Interaction, 8(2–3), 73–272.10.1561/1100000049Search in Google Scholar

[4] Bimber, O., & Raskar, R. (2005). Spatial augmented reality: Merging real and virtual worlds. CRC press.10.1201/b10624Search in Google Scholar

[5] Bonetti, F., Warnaby, G., & Quinn, L. (2018). Augmented Reality and Virtual Reality in Physical and Online Retailing: A Review, Synthesis and Research Agenda. In T. Jung & M. C. tom Dieck (Eds.), Augmented Reality and Virtual Reality: Empowering Human, Place and Business (pp. 119–132). Springer International Publishing.10.1007/978-3-319-64027-3_9Search in Google Scholar

[6] Brynjolfsson, E., McAfee, A., & Cummings, J. (2014). The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies (MP3 Una edition). Brilliance Audio.Search in Google Scholar

[7] Ciolek, T. M. (1983). The proxemics lexicon: A first approximation. Journal of Nonverbal Behavior, 8(1), 55–79.10.1007/BF00986330Search in Google Scholar

[8] Ciriello, R. F., Aschoff, F.-R., Dolata, M., & Richter, A. (2014). Communicating ideas purposefully-toward a design theory of innovation artifacts.Search in Google Scholar

[9] Dolata, M., Agotai, D., Schubiger, S., & Schwabe, G. (2019). Pen-and-paper Rituals in Service Interaction: Combining High-touch and High-tech in Financial Advisory Encounters. Proc. ACM Hum.-Comput. Interact., 3(CSCW), 224:1–224:24.10.1145/3359326Search in Google Scholar

[10] Dolata, M., Agotai, D., Schubiger, S., & Schwabe, G. (2020). Advisory Service Support that Works: Enhancing Service Quality with a Mixed-reality System. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 120:1–120:22.10.1145/3415191Search in Google Scholar

[11] Dolata, M., Comes, T., Schenk, B., & Schwabe, G. (2016). Persuasive Practices: Learning from Home Security Advisory Services. In Proc. Intl. Conf. Persuasive Technology – PERSUASIVE (pp. 176–188).10.1007/978-3-319-31510-2_15Search in Google Scholar

[12] Dolata, M., Kilic, M., & Schwabe, G. (2019, January 8). When a computer speaks institutional talk: Exploring challenges and potentials of virtual assistants in face-to-face advisory services. In Proc. Hawaii Intl. Conf. on System Sciences. Hawaii Intl. Conf. on System Sciences.10.24251/HICSS.2019.014Search in Google Scholar

[13] Dolata, M., Kilic, M., & Schwabe, G. (2015). Unpacking the Artifact Knowledge: Secondary Data Analysis in Design Science Research. In Proceedings of the International Conference on Design Science Research in Information Systems and Technology (pp. 327–342).10.1007/978-3-319-18714-3_21Search in Google Scholar

[14] Dolata, M., Schenk, B., Fuhrer, J., Marti, A., & Schwabe, G. (2020). When the system does not fit: Coping strategies of employment consultants. Computer Supported Cooperative Work, 29(6), 657–696.10.1007/s10606-020-09377-xSearch in Google Scholar

[15] Dolata, M., & Schwabe, G. (2016a). Design Thinking in IS Research Projects. In W. Brenner & F. Uebernickel (Eds.), Design Thinking for Innovation (pp. 67–83). Springer International Publishing.10.1007/978-3-319-26100-3_5Search in Google Scholar

[16] Dolata, M., & Schwabe, G. (2016b). More interactivity with IT support in advisory service encounters? In Proc. Conf. Mensch und Computer.Search in Google Scholar

[17] Dolata, M., & Schwabe, G. (2017a). Involvement Practices in Persuasive Service Encounter: The Case of Home Security Advice. In Proc. European Conf. on Information Systems.Search in Google Scholar

[18] Dolata, M., & Schwabe, G. (2017b). Tuning in to More Interactivity – Learning from IT Support for Advisory Service Encounters. I-Com: Journal of Interactive Media, 16(1), 23–33.10.1515/icom-2016-0042Search in Google Scholar

[19] Dolata, M., & Schwabe, G. (2017c). Paper Practices in Institutional Talk: How Financial Advisors Impress their Clients. Computer Supported Cooperative Work (CSCW), 769–805.10.1007/s10606-017-9279-8Search in Google Scholar

[20] Dolata, M., & Schwabe, G. (2018). Don’t be afraid! Persuasive Practices in the Wild. Computer Supported Cooperative Work (CSCW), 27(1), 427–462.10.1007/s10606-018-9330-4Search in Google Scholar

[21] Dolata, M., Steigler, S., Nüesch, F., Schock, U., Agotai, D., Schubiger, S., Kilic, M., & Schwabe, G. (2019). Welcome, computer! How do participants introduce a collaborative application during face-to-face interaction? In Proc. Intl. Conf. on Human-Computer Interaction – INTERACT.10.1007/978-3-030-29387-1_35Search in Google Scholar

[22] Dourish, P. (2001). Where the action is: The foundations of embodied interaction. MIT Press.10.7551/mitpress/7221.001.0001Search in Google Scholar

[23] Dourish, P., & Bellotti, V. (1992). Awareness and Coordination in Shared Workspaces. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work (pp. 107–114).10.1145/143457.143468Search in Google Scholar

[24] Drew, P., & Heritage, J. (1992). Analyzing talk at work: An introduction. In P. Drew & J. Heritage (Eds.). Talk at work: Interaction in institutional settings. Cambridge Univ. Press.Search in Google Scholar

[25] Eppler, M. J., & Bresciani, S. (2013). Visualization in management: From communication to collaboration. A response to Zhang. Journal of Visual Languages & Computing, 24(2), 146–149.10.1016/j.jvlc.2012.11.003Search in Google Scholar

[26] Färber, A., Zigan, N., Dolata, M., Stalder, P., Koppitz, A., & Schwabe, G. (2019, January 8). The Digital Transformation of Physician–Patient Consultations: Identifying Problems and Approaches to Improve Adherence.10.24251/HICSS.2019.500Search in Google Scholar

[27] Fischer, J. E., Crabtree, A., Colley, J. A., Rodden, T., & Costanza, E. (2017). Data work: How energy advisors and clients make IoT data accountable. Computer Supported Cooperative Work (CSCW), 1–30.10.1007/s10606-017-9293-xSearch in Google Scholar PubMed PubMed Central

[28] Frandsen, A.-C., Hiller, T. B., Traflet, J., & McGoun, E. G. (2013). From money storage to money store: Openness and transparency in bank architecture. Business History, 55(5), 695–720.10.1080/00076791.2012.715282Search in Google Scholar

[29] Gaver, W. (2012). What should we expect from research through design? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 937–946).10.1145/2207676.2208538Search in Google Scholar

[30] Giesbrecht, T., Comes, T., & Schwabe, G. (2015). Back in Sight, Back in Mind: Picture-Centric Support for Mobile Counseling Sessions. In Proc. Conf. Computer Supported Cooperative Work.10.1145/2675133.2675169Search in Google Scholar

[31] Giesbrecht, T., Schenk, B., & Schwabe, G. (2015). Empowering front office employees with counseling affordances. Transforming Government: People, Process and Policy, 9(4), 517–544.10.1108/TG-02-2015-0006Search in Google Scholar

[32] Giesbrecht, T., Schwabe, G., & Schenk, B. (2016). Service encounter thinklets: How to empower service agents to put value co-creation into practice. Information Systems Journal, 27(2), 171–196.10.1111/isj.12099Search in Google Scholar

[33] Harvey, A. C. (1990). Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press.10.1017/CBO9781107049994Search in Google Scholar

[34] Heinrich, P., Kilic, M., Aschoff, F.-R., & Schwabe, G. (2014). Enabling relationship building in tabletop-supported advisory settings. In Proc. Conf. Computer Supported Cooperative Work and Social Computing (pp. 171–183).10.1145/2531602.2531697Search in Google Scholar

[35] Heinrich, P., Kilic, M., & Schwabe, G. (2014, December 15). Microworlds as the locus of consumer education in financial advisory services. In Proc. Intl. Conf. on Information Systems.Search in Google Scholar

[36] Heinrich, P., & Schwabe, G. (2018). Facilitating Informed Decision-Making in Financial Service Encounters. Business & Information Systems Engineering, 60(4), 317–329.10.1007/s12599-017-0501-5Search in Google Scholar

[37] Heritage, J., & Clayman, S. (2010). Talk in action: Interactions, identities, and institutions. Wiley-Blackwell.10.1002/9781444318135Search in Google Scholar

[38] Heritage, J., & Greatbatch, D. (1989). On the institutional character of institutional talk: The case of news interviews. In Discourse in Professional and Everyday Culture. Linko Ping, Department of Communication Studies, University of Linko Ping, Sweden (pp. 47–98).Search in Google Scholar

[39] Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design Science in Information Systems Research. MIS Quarterly, 28(1), 75–105.10.2307/25148625Search in Google Scholar

[40] Heyman, S., & Artman, H. (2015). Computer Support for Financial Advisors and Their Clients: Co-creating an Investment Plan. In Proc. Conf. Computer Supported Cooperative Work and Social Computing (pp. 1313–1323).10.1145/2675133.2675236Search in Google Scholar

[41] Holman, D., & Vertegaal, R. (2008). Organic User Interfaces: Designing Computers in Any Way, Shape, or Form. Commun. ACM, 51(6), 48–55.10.1145/1349026.1349037Search in Google Scholar

[42] Holman, D., Vertegaal, R., Altosaar, M., Troje, N., & Johns, D. (2005). Paper windows: Interaction techniques for digital paper. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 591–599).10.1145/1054972.1055054Search in Google Scholar

[43] Hornecker, E., & Buur, J. (2006). Getting a Grip on Tangible Interaction: A Framework on Physical Space and Social Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 437–446).10.1145/1124772.1124838Search in Google Scholar

[44] Ishii, H. (2008). The Tangible User Interface and Its Evolution. Commun. ACM, 51(6), 32–36.10.1145/1349026.1349034Search in Google Scholar

[45] Jungermann, H., & Fischer, K. (2005). Using expertise and experience for giving and taking advice. In The Routines of Decision Making (pp. 157–173).Search in Google Scholar

[46] Kendon, A. (1977). Studies in the behavior of social interaction (Vol. 6). Humanities Press Intl.Search in Google Scholar

[47] Kendon, A. (2010). Spacing and Orientation in Co-present Interaction. In Development of Multimodal Interfaces: Active Listening and Synchrony (pp. 1–15). Springer, Berlin, Heidelberg.10.1007/978-3-642-12397-9_1Search in Google Scholar

[48] Khan, M., Trujano, F., Choudhury, A., & Maes, P. (2018). Mathland: Playful Mathematical Learning in Mixed Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (pp. D108:1–D108:4).10.1145/3170427.3186499Search in Google Scholar

[49] Kilic, M. (2018). IT-unterstützte Bedarfserhebung in der Finanzberatung (p. 294) [PhD Dissertation, University of Zurich].Search in Google Scholar

[50] Kilic, M., Dolata, M., & Schwabe, G. (2017). Why do you ask all those questions? Supporting client profiling in financial service encounters. In Proc. Hawaii Intl. Conf. System Sciences.10.24251/HICSS.2017.572Search in Google Scholar

[51] Kilic, M., Dolata, M., & Schwabe, G. (2016). How IT-Artifacts Disturb Advice Giving – Insights from Analyzing Implicit Communication. In Proc. Hawaii Intl. Conf. System Sciences (pp. 878–887).10.1109/HICSS.2016.113Search in Google Scholar

[52] Kilic, M., Heinrich, P., & Schwabe, G. (2015). Coercing into Completeness in Financial Advisory Service Encounters. In Proc. Intl. Conf. Computer-Supported Cooperative Work and Social Computing (pp. 1324–1335).10.1145/2675133.2675289Search in Google Scholar

[53] Koskinen, I., Zimmerman, J., Binder, T., Redstrom, J., & Wensveen, S. (2013). Design Research Through Practice: From the Lab, Field, and Showroom. IEEE Transactions on Professional Communication, 56(3), 262–263.10.1109/TPC.2013.2274109Search in Google Scholar

[54] Lewe, H., & Krcmar, H. (1991). The Design Process for a Computer-Supported Cooperative Work Research Laboratory: The Hohenheim CA Team Room. Journal of Management Information Systems, 8(3), 69–85.10.1080/07421222.1991.11517930Search in Google Scholar

[55] Luff, P., Pitsch, K., Heath, C., Herdman, P., & Wood, J. (2009). Swiping paper: The second hand, mundane artifacts, gesture and collaboration. Personal and Ubiquitous Computing, 14(3), 287–299.10.1007/s00779-009-0253-4Search in Google Scholar

[56] Macintosh, G. (2007). Customer orientation, relationship quality, and relational benefits to the firm. Journal of Services Marketing, 21(3), 150–159.10.1108/08876040710746516Search in Google Scholar

[57] Mittleman, D. (2009). Planning and Design Considerations for Computer Supported Collaboration Spaces. Journal of the Association for Information Systems, 10(3).10.17705/1jais.00185Search in Google Scholar

[58] Novak, J., & Schmidt, S. (2009). When Joy Matters: The Importance of Hedonic Stimulation in Collocated Collaboration with Large-Displays. In T. Gross, J. Gulliksen, P. Kotzé, L. Oestreicher, P. Palanque, R. O. Prates, & M. Winckler (Eds.), Human-Computer Interaction – INTERACT 2009 (pp. 618–629). Springer Berlin Heidelberg.10.1007/978-3-642-03658-3_66Search in Google Scholar

[59] Novak, J., & Schwabe, G. (2009). Designing for reintermediation in the brick-and-mortar world: Towards the travel agency of the future. Electronic Markets, 19(1), 15–29.10.1007/s12525-009-0003-5Search in Google Scholar

[60] Nueesch, R., Zerndt, T., Alt, R., & Ferretti, R. G. (2016). Tablets Penetrate the Customer Advisory Process: A Case from a Swiss Private Bank. In BLED Proc. (p. 18).Search in Google Scholar

[61] Nunamaker, J. F., Dennis, A. R., Valacich, J. S., Vogel, D., & George, J. F. (1991). Electronic Meeting Systems. Commun. ACM, 34(7), 40–61.10.1145/105783.105793Search in Google Scholar

[62] Nussbaumer, P., Matter, I., & Schwabe, G. (2012). “Enforced” vs. “Casual” Transparency—Findings from IT-Supported Financial Advisory Encounters. ACM Trans. Management Information Systems, 3(2), 11:1–11:19.10.1145/2229156.2229161Search in Google Scholar

[63] Oehler, A., & Kohlert, D. (2009). Financial Advice Giving and Taking—Where are the Market’s Self-healing Powers and a Functioning Legal Framework When We Need Them? Journal of Consumer Policy, 32(2), 91–116.10.1007/s10603-009-9099-4Search in Google Scholar

[64] Oehler, A., Kohlert, D., Jungermann, H., Reisch, L., & Micklitz, H.-W. (2010). The quality of financial investment advice for private investors: Problems in the advice process and potential solutions. Statement by the Scientific Advisory Council on Consumer and Food Policy at the Federal Ministry for Food.Search in Google Scholar

[65] Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, 24(3), 45–77.10.2753/MIS0742-1222240302Search in Google Scholar

[66] Pinch, T., & Clark, C. (1986). The Hard Sell: ‘Patter Merchanting’ and the Strategic (Re)Production and Local Management of Economic Reasoning in the Sales Routines of Markt Pitchers. Sociology, 20(2), 169–191.10.1177/0038038586020002002Search in Google Scholar

[67] Prante, T., Streitz, N. A., & Tandler, P. (2004). Roomware: Computers disappear and interaction evolves. Computer, 37(12), 47–54.10.1109/MC.2004.255Search in Google Scholar

[68] Ramsey, R. P., & Sohi, R. S. (1997). Listening to your customers: The impact of perceived salesperson listening behavior on relationship outcomes. Journal of the Academy of Marketing Science, 25(2), 127–137.10.1007/BF02894348Search in Google Scholar

[69] Rosson, M. B., & Carroll, J. M. (2002). Usability engineering: Scenario-based development of human-computer interaction. Morgan Kaufmann Publishers Inc.Search in Google Scholar

[70] Samuels, J. T., Mathew, A. P., Kommanivanh, C., Kwon, D., Gomez, L., Thunder, B. A., Velazquez, D., Martinez, M., & LaQueens, L. (2018). Art, Human Computer Interaction, and Shared Experiences: A Gun Violence Prevention Intervention. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (pp. D404:1–D404:4).10.1145/3170427.3186526Search in Google Scholar

[71] Schenk, B., Dolata, M., Schwabe, C., & Schwabe, G. (2021). What citizens experience and how omni-channel could help–insights from a building permit case. Information Technology & People, in Google Scholar

[72] Schmidt-Rauch, S., & Nussbaumer, P. (2011). Putting value co-creation into practice: A case for advisory support. In Proc. of European Conference on Information Systems – ECIS. European Conference on Information Systems, Helsinki.Search in Google Scholar

[73] Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., & Pea, R. D. (2015). 3D tangibles facilitate joint visual attention in dyads. Proceedings of 11th International Conference of Computer Supported Collaborative Learning, 1, 156–165.Search in Google Scholar

[74] Schwabe, G. (2002). (T)Räume der Zusammenarbeit: Probleme und neue Ansätze der Sitzungsunterstützung (Rooms of Cooperation: Problems and New Approaches in Session Support). I-Com: Journal of Interactive Media 1(1), 4.10.1524/icom.2002.1.1.004Search in Google Scholar

[75] Schwabe, G., & Nussbaumer, P. (2009). Why information technology is not being used for financial advisory. In Proc. European Conf. on Information Systems.Search in Google Scholar

[76] Sommers, C., & Jenkins, D. (2013). Whiteboard Selling: Empowering Sales Through Visuals (1.). Wiley.10.1002/9781118461587Search in Google Scholar

[77] Stadlbauer, A. (2010). Pencil Selling: Symbole zeichnen – sekundenschnell und professionell (1. Aufl). Trauner.Search in Google Scholar

[78] Steimle, J. (2012). Pen-and-Paper User Interfaces. Springer Berlin Heidelberg.10.1007/978-3-642-20276-6Search in Google Scholar

[79] Steimle, J. (2009). Designing pen-and-paper user interfaces for interaction with documents. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (pp. 197–204).10.1145/1517664.1517707Search in Google Scholar

[80] Streitz, N. A., Geißler, J., & Holmer, T. (1998). Roomware for Cooperative Buildings: Integrated Design of Architectural Spaces and Information Spaces. In N. A. Streitz, S. Konomi, & H.-J. Burkhardt (Eds.), Cooperative Buildings: Integrating Information, Organization, and Architecture (pp. 4–21). Springer.10.1007/3-540-69706-3_3Search in Google Scholar

[81] Svinhufvud, K., & Vehviläinen, S. (2013). Papers, documents, and the opening of an academic supervision encounter. Text & Talk, 33(1), 139–166.10.1515/text-2013-0007Search in Google Scholar

[82] Verhallen, T. M. M., Greve, H., & Frambach, R. Th. (1997). Consultative selling in financial services: An observational study of the mortgage mediation process. Intl. J. Bank Marketing, 15(2), 54–59.10.1108/02652329710160466Search in Google Scholar

[83] Ware, C. (2013). Perceiving complex causation through interaction. In Proceedings of the Symposium on Computational Aesthetics (pp. 29–35).10.1145/2487276.2487279Search in Google Scholar

[84] Weiser, M., & Brown, J. S. (1996). Designing calm technology. PowerGrid Journal, 1(1), 75–85.10.1007/978-1-4612-0685-9_6Search in Google Scholar

[85] Withgott, M. (2015). What Do Objects Mean?: Early Tangibility in and Around Interval Research. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 5–12).10.1145/2677199.2680556Search in Google Scholar

[86] Xiao, R., Hudson, S., & Harrison, C. (2016). DIRECT: Making Touch Tracking on Ordinary Surfaces Practical with Hybrid Depth-Infrared Sensing. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (pp. 85–94).10.1145/2992154.2992173Search in Google Scholar

[87] Zimmerman, J., & Forlizzi, J. (2014). Research Through Design in HCI. In J. S. Olson & W. A. Kellogg (Eds.), Ways of Knowing in HCI (pp. 167–189). Springer New York.10.1007/978-1-4939-0378-8_8Search in Google Scholar

[88] Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 493–502).10.1145/1240624.1240704Search in Google Scholar

[89] Zuckerman, O., & Gal-Oz, A. (2013). To TUI or not to TUI: Evaluating performance and preference in tangible vs. graphical user interfaces. International Journal of Human-Computer Studies, 71(7), 803–820.10.1016/j.ijhcs.2013.04.003Search in Google Scholar

Published Online: 2021-11-27
Published in Print: 2021-12-20

© 2021 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 29.11.2023 from
Scroll to top button