Skip to content
BY-NC-ND 3.0 license Open Access Published by De Gruyter September 12, 2015

Use cases and usability challenges for head-mounted displays in healthcare

  • T. Mentler EMAIL logo , C. Wolters and M. Herczeg


In the healthcare domain, head-mounted displays (HMDs) with augmented reality (AR) modalities have been reconsidered for application as a result of commercially available products and the needs for using computers in mobile context. Within a user-centered design approach, interviews were conducted with physicians, nursing staff and members of emergency medical services. Additionally practitioners were involved in evaluating two different head-mounted displays. Based on these measures, use cases and usability considerations according to interaction design and information visualization were derived and are described in this contribution.

1 Introduction

Head-mounted displays (HMDs) have been discussed since the 1960s [1] but actually deployed in only a few domains like military applications. A renewed interest in HMDs with augmented reality (AR) modalities is fostered by commercially available and cheaper products and the needs for using computers in mobile contexts, among others in the healthcare domain [2].

As a starting point for a user-centered design process and in order to derive relevant scenarios, we reviewed current literature and conducted interviews with more than 20 physicians, nursing staff and members of emergency medical services. Two HMDs (Google Glass, Epson Moverio BT-200) have been evaluated for use in the medical domain.

Google Glass (see Figure 1) can be controlled by either using a touch pad at the right sidepiece or a fixed and small set of voice commands. Its display is shown on the right side of the user’s visual field and equals a 64 cm (~25”) high definition screen seen from a distance of 250 cm (~8 ft.). The device should be positioned slightly above and not directly in front of the eye [3].

Figure 1 Google Glass
Figure 1

Google Glass

Another device, the Epson Moverio BT-200 (see Figure 2), features a binocular display which allows viewing contents centered like watching a 200 cm display (~80”) seen from a distance of 500 cm (~16 ft.). A trackpad can be used for manually controlling applications and builtin motion trackers and a camera can be used for hands-free interactions. Location-based services can be realized through an integrated GPS-sensor [4].

Figure 2 Epson Moverio BT-200
Figure 2

Epson Moverio BT-200

Our research indicates great interest in HMDs with AR capabilities by medical professionals all along the rescue and medical chain, independent from specific device models. Use cases mentioned spin around two aspects: first using an interactive system hands-free and second having medical information ubiquitously available. However, there is still uncertainty among the medical experts about special features, limitations and reliability of HMDs. Most of the participants had no practical experience with an HMD and just got impressions and ideas from advertisements or magazines.

2 Use cases for HMDs

In the following sections, envisioned use cases of HMDs in pre-hospital medical care, medical treatment in hospitals and nursing care in clinical environments will be outlined. Following up, some major usability challenges will be described (see section 3).

2.1 Pre-hospital medical care

With respect to Emergency Medical Services (EMS) and out-of-hospital care, HMDs are primarily related to disaster medicine and mass casualty incidents (MCIs). Such an event “[...] generates more patients at one time than locally available resources can manage using routine procedures. It therefore requires exceptional emergency arrangements and additional or extraordinary assistance” [5]. In this regard, computer-supported triage based on algorithms and software wizards is one of the thematic priorities and has already been applied to HMDs, e. g. in connection with Quick Response (QR) codes or multimodal interfaces for physicians acting remotely [6, 7]. Telemedicine is seen to be a use case for HMDs in regular transport and emergency missions [8].

Apart from supporting the triage process, the experts interviewed by us mentioned further ideas about using HMDs in MCIs. By logging actions and communication of EMS members wearing HMDs, debriefings and evaluations could be facilitated more easily. Attention should be paid to the fact that HMDs are considered an additional source of information and not as a replacement for current tools.

Image recognition features of HMDs could be used to automatically identify signs posted for hazardous materials and environments. In this regards, further steps could be marking high-risk areas and displaying recommended modes of behaviour. Such feature would even contribute to a better collaboration of EMS and firefighters. The latter are sometimes unsatisfied with vague feedback about hazard potentials at the site of the accident given by EMS members.

Furthermore, looking for the location of a suitable hospital or sharing perspectives with co-workers have been named as use cases which could be realized with the aid of HMDs. However, they haven’t been described in detail and need to be elaborated [9].

2.2 Medical treatment in hospitals

As with pre-hospital medical care, telemedicine and video-based reviews of one’s own and others work are often mentioned use cases for HMDs in clinical contexts. Specialist located in another department could be consulted on-thefly and surgical trainees could reflect upon their work during residency [10].

Teaching in general is an important aspect while considering HMDs in hospital environments. Although there is uncertainty whether HMDs could be a milestone in surgical education or just a gadget; some physicians have already made plans to stream their surgeries to students [11, 12].

In addition, HMDs could be used to display further information about a patient or treatment on the spot. For example [13]: “A cardiologist in a cath lab overlays the fluoroscopy as they perform a femoral catheterization for a patient with a recent myocardial infarct”.

Finally, HMDs could contribute to the breakthrough of electronic health records (EHR). The medical staff in charge could access and update patient-related data standing by the bedside. Data from different medical devices could be integrated, aggregated and summarized. Likewise, they could be informed about incoming laboratory results without being interrupted by phone calls or messengers.

2.3 Nursing care in hospitals

Apart from medical treatment, nursing care is important for patients’ well-being. Nursing staff often has to care for several patients simultaneously during their shifts.

As with medical treatment, HMDs could enable telemedicine in the context of nursing care. Physicians could support nursing staff in extraordinary circumstances, e.g. if a patient suddenly collapses at night with no other doctor available immediately.

As part of handover, caregivers could share information about patient-specific occurrences asynchronously and without the need to remember details after several hours of work. Important hints could even be displayed, while entering the patient’s room for the first time in a certain shift.

Finally, the before-mentioned uses case according to EHRs can be applied to nursing records as well. They take up a lot of time and can often only be edited at the nursing station at the present time.

3 Usability challenges

Although usability concerns were rarely expressed by the interviewed medical experts, interaction design and information visualization for HMDs can be expected turning out to be major challenges, especially with respect to critical contexts of use. They will be outlined in the following sections.

3.1 Interaction design

Hands-free interaction is one of the greatest advantages medical experts associate with HMDs. As described before, possibilities and details of interaction design depend on the specific device. Nevertheless, two basic approaches can be distinguished currently – speech recognition and touch gestures. While the first will be difficult in noisy environments of medical care, the latter might interfere with hygiene requirements and requires remembering gestures.

Gesture control without touching can be realized with additional devices like an interactive wristband [14]. However, such solutions will increase the complexity of systems and require interaction solutions which fit body or arm movement to screen transitions and other aspects of graphical user interfaces.

Classical approaches like Direct Manipulation [15] cannot be applied straightforward. New concepts for Cross-Device Interaction (XDI), as “the type of interaction, where human users interact with multiple separate input and output devices, where input devices will be used to manipulate content on output devices within a perceived interaction space with immediate and explicit feedback” [16], might be more helpful.

3.2 Information visualization

As with the interaction design, details about visualizing patient-related data depend on the specific display device. In general, there will be trade-offs between the amount of data displayed, its legibility and the available field of view for perceiving surrounding information.

Clinical or hospital information systems (CIS/HIS) store huge amounts of data and even single EHRs have to be managed with advanced applications which often lack of usability [17]. Limited screen sizes and different interaction modalities of HMDs require even more specialized design solutions. While browsing datasets with the help of search and filter function is at least possible, but not necessarily efficient, with applications developed for one or more computer monitors, mouse and keyboard, this would not work with HMDs.

Location-based and context-aware services could be a solution to the aforementioned challenges but have to be critically judged as well. Accuracy of GPS within buildings is often difficult, at least if there is the need to distinguish between patients lying in neighbouring beds.

Augmenting reality in safety- and time-critical contexts like medical care requires precise alignment of virtual and physically present objects. Patient’s movements and other obstacles (e.g. partially hidden elements) need to be carefully considered.

3.3 Context of use

No matter what technical specification an interactive system has, its usability depends on the suitability for the context of use. HMDs have to be aligned with current work environments, workflows and tools. Otherwise health professionals who are already dealing with a variety of desktop PCs, portable devices (e.g. tablet PCs), smartphones and medical devices will have to master another kind of interactive technology – the HMD.

With respect to pre-hospital medical care we refer to the “Care&Prepare-Principle”. It states that an interactive system for handling MCIs should be a “natural” extension of an application for managing regular rescue und transport missions [18]. Therefore, telemedicine might be promising.

Legal aspects concerning privacy and security have to be addressed. Furthermore, possibilities to misuse the HMD, e.g. by taking images and posting them into unauthorized cloud-space, have to be identified and averted.

HMDs have to be accepted not only by medical professionals but by patients as well. Otherwise physician-patient relations might be disturbed right in the moment the physician is entering the room to the detriment of both groups.

4 Conclusion

HMDs, in this case understood as a generic term for wearable devices augmenting reality, are of current interest in healthcare. Usability challenges concerning interaction design and information visualization have to be solved in order to develop usable solutions. Due to the different hardware approaches [19], it is at least questionable, whether general patterns and best practices can be derived at this time. Practical solutions or Cross-Device Interaction are just emerging. Prototypes have to be evaluated in field studies and with respect to all aspects of the context of use. Therefore, all stakeholders have to be involved – health-care professionals as well as patients.


The authors would like to thank Za-hide Okcu and Henrik Berndt for conducting interviews and field studies.

Author's Statement

  1. Conflict of interest: Authors state no conflict of interest. Material and Methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use has been complied with all the relevant national regulations, institutional policies and in accordance the tenets of the Helsinki Declaration, and has been approved by the authors’ institutional review board or equivalent committee.


[1] Sutherland IE. Head-mounted three dimensional display. In: Proceedings of the December 9-11, 1968, fall joint computer conference, part I (AFIPS ’68 (Fall, part I). New York: ACM 1968: 757-764.10.1145/1476589.1476686Search in Google Scholar

[2] Malvey D, Slovensky JD. MHealth: Transforming Healthcare. New York: Springer 2014.10.1007/978-1-4899-7457-0Search in Google Scholar

[3] Google Inc. Tech specs. Available online: in Google Scholar

[4] Seiko Epson Corporation. Techn. Daten. Available online: in Google Scholar

[5] World Health Organization. Mass casualty management systems: Strategies and guidelines for building health sector capacity. Available online: in Google Scholar

[6] Carenzo L, Bara FL, Ingrassia PL, Colombo D, Costa A, Della Corte F. Disaster medicine through Google Glass. Eur J Emerg Med. 2014.10.1097/MEJ.0000000000000229Search in Google Scholar PubMed

[7] Cicero MX, Walsh B, Solad Y, Whitfill T, Paesano G, Kim K, Baum CR, Cone DC. Do you see what I see? Insights from using google glass for disaster telemedicine triage. Prehospital and disaster medicine 2015; 30:4–8.10.1017/S1049023X1400140XSearch in Google Scholar PubMed

[8] Widmer A, Müller H. Using Google Glass to enhance pre-hospital care. Swiss Medical Informatics 2014.10.4414/smi.30.00316Search in Google Scholar

[9] Google Glass: Datenbrille für Rettungskräfte. Available online: in Google Scholar

[10] Paro JA, Nazareli R, Gurjala A, Berger A, Lee GK. Video-Based Self-Review: Comparing Google Glass and GoPro Technologies. Ann Plast Surg 2015.10.1097/SAP.0000000000000423Search in Google Scholar PubMed

[11] Waxman BP. Medicine in small doses. Google Glass: a new dimension in surgical education or just another gimmick? ANZ J Surg. 2014; 84:810.10.1111/ans.12841Search in Google Scholar PubMed

[12] Keen CE. Google Glass goggles stream live surgery. European Hospital 2014; 23: 1-2.Search in Google Scholar

[13] Aungst T. How Google Glass could revolutionize medicine. Available online: in Google Scholar

[14] Dementyev A, Paradiso JA. WristFlex: low-power gesture input with wrist-worn pressure sensors. In: Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST ’14). New York: ACM 2014:161-166.10.1145/2642918.2647396Search in Google Scholar

[15] Hutchins EL, Hollan JD, Norman, DA. Direct Manipulation Interfaces. In: Norman DA, Draper SW, editors. User Centered System Design. Hillsdale: Lawrence Erlbaum 1986: 87-124.10.1201/b15703-5Search in Google Scholar

[16] Scharf F, Wolters C, Herczeg M, Cassens J. Cross-Device Interaction: Definition, Taxonomy and Application. In Proceedings of AMBIENT 2013: The Third International Conference on Ambient Computing, Applications, Services and Technologies. Porto: IARIA 2013: 35-41.Search in Google Scholar

[17] HIMSS EHR Usability Task Force. Defining and Testing EMR Usability: Principles and Proposed Methods of EMR Usability Evaluation and Rating. Available online: in Google Scholar

[18] Mentler T, Herczeg M, Jent S, Stoislow M, Kindsmüller MC. Routine Mobile Applications for Emergency Medical Services in Mass Casualty Incidents. In: Biomed Tech - Proceedings BMT 2012. Vol. 57 (Suppl. 1). Walter de Gruyter 2012: 784-787.10.1515/bmt-2012-4457Search in Google Scholar PubMed

[19] Kompetenzzentrum Virtuelle Realität und Kooperatives Engineering w. V. - Virtual Dimension Center VDC. Whitepaper Head Mounted Displays & Datenbrillen: Einsatz und Systeme. Available online:{&}_Datenbrillen.pdf.2015Search in Google Scholar

Published Online: 2015-9-12
Published in Print: 2015-9-1

© 2015 by Walter de Gruyter GmbH, Berlin/Boston

This article is distributed under the terms of the Creative Commons Attribution Non-Commercial License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Downloaded on 27.2.2024 from
Scroll to top button