Ultrasound has been established as a diagnostic tool in a wide range of applications. But especially for beginners, the alignment of sectional images to patient’s spatial anatomy can be cumbersome. The examing physician has to switch his/her view between the screen and patient frequently, what impedes an undisturbed diagnostic. A direct view onto the patient’s anatomy while regarding ultrasound images may help to overcome unergonomic examination.
Augmented reality in medicine has been presented since the 1990s [1, 2]. Still solutions suffer from high cost, regarding to immense technical requirements, and unergonomic designs [3, 4]. To solve these issues an affordable augmented reality system using smart glasses was created, that displays a (virtual) ultrasound image beneath a (real) ultrasound transducer.
The used hardware consists of an ultrasound device (Esaote MyLab70 XVG) including a linear 2-D ultrasound transducer, a PC (Intel Core2Duo, 2.6 GHz CPU, Geforce GTX 560) and smart glasses (Epson Moverio BT-200) including a smartphone (TI OMAP 4460, 1.2 GHz, 1 GB RAM) and an built-in camera.
As illustrated in Figure 1 the PC collects raw data via Ethernet from the ultrasound device, recalculates the image data and holds these data available as byte arrays. Subsequent the smart phone collects the byte arrays via WLAN and reconstructs the ultrasound image. An integrated camera enables the smart glasses to collect images of the transducer and the optical marker on it. These images are sent to the smartphone, where the position of the optical marker can be tracked. The ultrasound images are scaled and, if necessary, rotated to the correct viewing angle. Finally the smart glasses display the (virtual) image at the marker position under the (real) ultrasound transducer (s. Figure 2).
The system needs no initial calibration on startup because the client knows the fixed marker position on the transducer. The image position and size is calculated from the position and size of the marker.
The server, running on the PC, was implemented using C#. The client on the smartphone consists of C#-scripts and an app, created with Unity3D and the Qualcomm Augmented Reality SDK “Vuforia”. While the scripts manage the communication between the server and the client, the app creates the ultrasound images, detects the optical marker and places the images with the calculated rotation and scale at the correct position.
The ultrasound images can be visualized inside the smart glasses as seen in Figure 2 at about 25 fps and a resolution of 128x256 pixels. Detecting the optical marker is possible within a viewing angle between about 70 and 110 degrees. When moving the transducer the image follows without noticeable delay.
4 Conclusion and outlook
Smart glasses can be used for ultrasound examinations in principle. Those systems are ergonomic and low cost due to minor technical requirements. The framerate and resolution is adequate for most applications. However the detection of the optical marker needs to be improved to realize higher viewing angles.
Current works determine to the implementation of faster data transfer algorithms to realize higher resolutions and framerates.
This work was funded by the Landesregierung NordrheinWestfalen in the Med in.NRW-program, grant no. GW01-078.
Y. Sato, M. Nakamoto, Y. Tamaki, T. Sasama, I. Sakita, Y. Nakajima, M. Monden and S. Tamura. Image Guidance of Breast Cancer Surgery Using 3-D Ultrasound Images and Augmented Reality Visualization. IEEE Transactions on Medical Imaging 1998; 17: 681-693. Google Scholar
M. Bajura, H. Fuchs and R. Ohbuchi. Merging Virtual Objects with the Real World: Seeing Ultrasound Imagery within the Patient. Computer Graphics 1992; 26: 203-210. Google Scholar
D. Magee, Y. Zhu, R. Ratnalingam, P. Gardner and D. Kessel. An augmented reality simulator for ultrasound guided needle placement training. Medical & Biological Engineering & Computing 2007; 45: 957-967. Google Scholar
M. Nakamoto, Y. Sato, M. Miyamoto, Y. Nakamjima, K. Konishi, M. Shimada, M. Hashizume and S. Tamura, “3D Ultrasound System Using a Magneto-optic Hybrid Tracker for Augmented Reality Visualization in Laparoscopic Liver Surgery,” Medical Image Computing and Computer-Assisted Intervention MICCAI 2002: 148-155 Google Scholar
About the article
Published Online: 2015-09-12
Published in Print: 2015-09-01
Conflict of interest: Authors state no conflict of interest. Material and Methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use has been complied with all the relevant national regulations, institutional policies and in accordance the tenets of the Helsinki Declaration, and has been approved by the authors’ institutional review board or equivalent committee.