Bioinformatics usually deals with biological macromolecules, the smallest entities of life, therefore, visualization plays an important role in this field. Stereoscopic rendering has been used for a long time for visualizing biological structures, dating back to the first hype of virtual reality technologies 25 years ago when the CAVE was invented and Head-Mounted Displays (HMD) became affordable. Triggered by the success of new, modern HMDs such as Oculus Rift and HTC Vive, Virtual Reality has recently regained a lot of attention. On the other hand, a new generation of large-scale “computational microscopes” is being established, such as the CAVE2, representing large, nearly 360 degree-spanning visualization facilities.
At the same time, advanced “Visual Analytics” solutions have become an important tool in all areas of scientific data analysis, combining visualization, data mining and analysis methods with appropriate user interaction. Uniting both VR and VA, “Immersive Analytics” now starts to make use of new technologies, immersing the scientist into the data, and, in the best case, enabling advanced insights. The present manuscript introduces a special issue on these very topics.
The special issue comprises six publications:
- Interactive Molecular Graphics for Augmented Reality using HoloLens (Section 4.1)
- Heuristic Modeling and 3D Stereoscopic Visualization of a Chlamydomonas reinhardtii Cell (Section 4.2)
- Molecular Dynamics Visualization (MDV): stereoscopic 3D display of biomolecular structure and interactions using the Unity game engine (Section 4.3)
- Mesoscopic rigid body modelling of the extracellular matrix self-assembly (Section 4.4)
- MinOmics, an integrative and immersive tool for multi-omics analysis (Section 4.5)
- Semantics for an integrative and immersive pipeline combining visualization and analysis of molecular data (Section 4.6)
1.1 A Workshop with Live and Immersive Demonstrations
The special session ’From Virtual Reality to Immersive Analytics in Bioinformatics’ was organized as part of two conferences: Stereoscopic displays and applications (SD&A) XXIX, taking place 29 Jan – 31 Jan 2018, and The engineering reality of virtual reality 2018 (ERVR), taking place 30 Jan – 1 Feb 2018, both in Burlingame, California as part of the 2018 IS&T International Symposium on Electronic Imaging – often just known as the Electronic Imaging Symposium.
Established in 1990, and now in its 29th year, SD&A focuses on developments covering the entire stereoscopic 3D imaging pipeline from capture, processing, and display, to perception. The conference brings together practitioners and researchers from industry and academia to facilitate an exchange of current information on stereoscopic imaging topics. The website is located at: http://stereoscopic.org.
The ERVR conference focuses on the creation of virtual and augmented reality systems. In addition to research in this area, the trend toward content building continues and practitioners find that technologies and disciplines must be tailored and integrated for specific visualization and interactive applications. This conference serves as a forum where advances and practical advice toward both creative activity and scientific investigation are presented and discussed. Research results as well as applications can be presented.
Most of the authors of the manuscripts presented in this contribution provided a live demonstration of their implementation at the conference venue. This demo provided a first hands experience and furthermore enabled us to gauge the public’s enthusiasm for such technologies, which was strong. Furthermore, videos of the implementations were shown, user feedback was described and use cases were reported.
Other manuscripts presented at the 2018 SD&A and ERVR conferences will be published in the Electronic Imaging Symposium conference proceedings available open access at: http://ist.publisher.ingentaconnect.com/content/ist/ei.
2 Virtual Reality in Bioinformatics
Virtual Reality-related topics play an important role in many research areas. In philosophy, Plato’s Allegory of the Cave – published in his famous opus Republic – discusses one of the basic principles in discriminating reality, projections and the way the human being relies on his senses to interpret the world. In this way, humans do not see reality, they perceive reality, relying on their senses. Where there is a common denominator for all human beings, there are, of course, strong individual differences, depending on the culture, socialization, and personal experiences.
Virtual Reality now extends the projections of the real world to the digital world, an important step in the quest for the ultimate display . Although the rise of new VR-related technology in the recent years is strongly affiliated to the HMDs such as Oculus Rift® and HTC VIVE® on the visualization side, and to the Leap Motion on the 3D interaction side , , , the ideas behind this technology are not new. The first HMD was developed by Sutherland around 1968 . One of the first approaches to involve multiple senses of a viewer was even developed a few years earlier by Morton Heilig . In the 1980s, the term Virtual Reality became popular, thanks to pioneers like Jaron Lanier, who developed first technological gadgets supporting the 3D visualization and interaction with the computer. The addition of haptics, as demonstrated for biological applications, was an important step . A big breakthrough of the 1990s was the invention of the CAVE®, the Cave Automatic Virtual Environment, a visualization system consisting of three to six stereoscopic projection screens, which were installed at many research facilities worldwide . 20 years later, the CAVE2®, a large, nearly 360°-spanning display environment , , and many other systems around the world are enabling room-scale and group experienced virtual reality. In parallel, the previously mentioned HMDs, such as Oculs Rift, were developed, and alternative devices like the Microsoft HoloLens® support Augmented Reality applications, enabling the projection of 3D scenes into the natural real world environment .
In Bioinformatics, many software applications were developed to support VR-related technologies. Obviously, the visualization of the smallest entities of life, not visible to the bare human eye, are one of the application cases where the use of VR-related technologies is reasonable and promising. Educational software like Meta!Blast, enabling the stereoscopic visualization of cell environments, goes a small step towards this direction . Bryson  provides a good overview of scientific applications of this 1990s era. Frederick Brooks’s paper  provides an inspiring general overview of this field from a personal perspective. Already in the early days of the CAVE, researchers started to visualize molecular structures in this environment . However, there was not a major breakthrough for this kind of scientific visualization, because they were limited to a few very cost-intensive research facilities. The arrival of new affordable technologies in recent years might be able to change this now.
3 Immersive Analytics in Bioinformatics
VR-related approaches are visually impressive. In the particular case of spatial structures they, furthermore, provide a precious extension of 3D visualizations to the bioinformatics area through their immersive character. However, they are often still far away from being very helpful in supporting the analytics process.
Visual Analytics was defined by Thomas and Cook as “the science of analytical reasoning facilitated by interactive visual interfaces” . To extend this approach to VR-related technology, the term Immersive Analytics was coined by Chandler et al.: “Immersive Analytics investigates how new interaction and display technologies can be used to support analytical reasoning and decision making“ . An important aspect of Immersive Analytics is usually the stereoscopic visualization to immerse the user into the environment . Furthermore, Immersive Analytics extends these approaches to multiple senses, haptic as well as olfactory feedback are potential elements of these approaches.
Although the term is quite new, related approaches were already developed many years ago. Software applications like MetNetVR integrated complex metabolic pathways into abstract cell environments and visualized them in a CAVE for analytical purposes around 2006 . For navigation purposes, a tablet was used. Using the CELLmicrocosmos Pathway Integration, metabolic pathways can be spatially embedded into virtual cell environments and explored in different virtual environments, such as zSpace® and/or CAVE2® , . The zSpace® is a semi-immersive stereoscopic display device which can be used for head-tracked stereoscopic visualization, enabling the display perspective to be adjusted as the user moves in front of the display. 3D interaction is possible with a special stylus pen. This setup enables hybrid-dimensional visualization and interaction, visualizing spatial structures with abstract data side-by-side . Aquaria, a molecular graphics and information system which can be used via a web browser to explore protein structures of the Protein Data Bank on different semantic layers, was extended by a Leap motion-based control kit which can be used to navigate the 3D models by hand , . In this way, it unites 3D navigation with hybrid-dimensional visualization, combining 3D protein structures with 2D representations of their primary structure.
These are only a few examples and a full review of Bioinformatics-related approaches in the field of Immersive Analytics is not within the scope of this publication. This special issue is intended to reveal that Immersive Analytics bears much potential for Bioinformatics.
4 From Virtual Reality to Immersive Analytics in Bioinformatics
The special session at SD&A 2018 we report on included six contributions providing a rich and varied overview of different directions VR-related research in the field of Bioinformatics is taking these days. We tried to order the approaches according to their progression along the path from Virtual Reality to Immersive Analytics. Therefore, the publications listed towards the end increasingly incorporate analytical elements into virtual environments.
4.1 Interactive Molecular Graphics for Augmented Reality Using HoloLens
An important device in the field of Augmented Reality applications is the Microsoft HoloLens®. This device can be used to place virtual objects into real-world environments. The manuscript shows proteins floating over a table or projected in front of a tree. In this way, the observer can inspect the spatial structure of the protein from all perspectives using natural displacement and body movement. Similar technologies may be used in future approaches to elaborate protein structures collaboratively in the context of a regular environment, liberated from the limited perspective provided by regular computer displays. In case of the HoloLens, the big advantage of an integrated computing device comes with a drawback: the compute performance is quite low. In case of molecular data, where usually thousands to millions of atoms are visualized as spheres, this limitation is problematic. In this work, the performance of the HoloLens is analyzed, extended by a good overview concerning related application cases. A natively developed Universal Windows Platform application with different rendering techniques is compared to a Unity application in terms of performance, varying the number of atoms to be visualized .
4.2 Heuristic Modeling and 3D Stereoscopic Visualization of a Chlamydomonas reinhardtii Cell
Chlamydomonas reinhardtii is an unicellular organism which plays an important role in the field of metabolic engineering as it can be used as an alternative energy source. In this work a spatial mesoscopic model of this cell was created using an interpretative cell visualization approach. The idea is to combine the knowledge of different information sources – such as publications, microscopic images, database information – with the purpose to create a well-evaluated cell model which can be used for visualization and exploration purposes. The so-created model was used as a basis for a stereoscopic 3D cell animation, discussing the underlying metabolic processes to extract energy from a biological cell. The publication discusses the nature of the different information sources, the design choices involved in the process (e.g. in terms of coloring, texturing and structure building), the way the flagella-based movement of the cell was animated, as well as the special requirements to create a stereoscopic 3D animation. The underlying observations are relevant for content creation for virtual environments and should encourage visualization researchers to use stereoscopic 3D visualization in the context of their work which also plays an important role in Virtual Reality-related applications .
4.3 Molecular Dynamics Visualization (MDV): Stereoscopic 3D Display of Biomolecular Structure and Interactions Using the Unity Game Engine
An important field in Bioinformatics as well as Chemoinformatics research is the simulation of molecular structures. One of the most established techniques is the molecular dynamics (MD) simulation approach. Such simulations were performed in the context of this work with the Gromacs  software. The visualization of molecular dynamics has a long tradition, using standard tools such as VMD or PyMol. However, these visualization packages are not optimized to be used in the context of multiscale virtual 3D stereoscopic environments. The molecular dynamics visualization (MDV) software has been developed on top of the Unity development platform and can be used with a range of large-scale immersive displays. The HIVE Cylinder at Curtin University for example consists of a 3 m high × 8 m diameter screen with a 180° field-of-view. The screen is illuminated using three 1920 × 1200 DLP projectors warped and blended to provide a continuous image across the screen, supporting 3D stereoscopic visualization with (active) liquid-crystal shutter 3D glasses. Using MDV, MD simulations can be collaboratively as well as immersively analyzed with groups of up to 50 people. Standard molecular visualization metaphors, such as ball & stick and secondary structure representations are implemented and can be selected via a 3D user interface .
4.4 Mesoscopic Rigid Body Modelling of the Extracellular Matrix Self-Assembly
As the structure of the extracellular matrix (ECM) is not well known due to technological limitations, alternative approaches are required to predict and model its structural properties. Using the game and physics engines of Unity 3D®, a framework was developed to model large molecules as dynamic chains of interacting rigid bodies and to observe self-assembly processes. Thereby, the ECM components can be modeled and evaluated at the mesoscopic scale. Properties like fibre flexibility as well as intermolecular interaction types and occurrences can be analyzed. Using an HMD, the user of this framework is immersed into the spatial simulation which can be easily explored by using the 3D controllers of the HMD. For this purpose, a gaming machine like device was placed into a virtual environment which can be triggered to run the ECM simulation. It is possible to observe the structural changes of the different ECM components and compare their structures to each other. The publication further compares the rendering and CPU performance of rigid bodies representing one to three million atoms .
4.5 MinOmics, an Integrative and Immersive Tool for Multi-omics Analysis
MinOmics is an integrated analysis pipeline and visualization framework for multi-omics analysis. In this work, a stereoscopic 25 MPixel display wall was used to display proteomic data on 1417 proteins of Chlamydomonas reinhardtii. For this purpose, UnityMol was used, a Unity 3D®-based software application which visualizes proteins in 3D . A number of hardware setups are discussed which could be used with MinOmics: a UnityMol-based WebGL and WebVR application using a regular monitor, a split-screen approach combined to a stereoscopic monitor, and a full-immersive environment using head-mounted displays. A number of scientific application cases for these setups are discussed. For example, the MinOmics framework can be used to explore multiple omics datasets. The authors discuss the exploration of redox proteomic datasets in the context of Chlamydomonas reinhardtii for glutathionylation, nitrosylation and disulphide bonds. MinOmics can be used to explore protein specificity of multiple redox post translational modifications (PTM), the cysteine specificity of multiple redox PTMs, and to explore structural determinants of redox-modified cysteines .
4.6 Semantics for an Integrative and Immersive Pipeline Combining Visualization and Analysis of Molecular Data
Most bioinformatics-related workflows require the use of a number of different tools. Often this diversity means that data has to be transformed, converted to other formats and transferred leading to a huge overhead of time and costs. This overhead applies especially to virtual environments, where sometimes the data to be immersively visualized has to be optimized by visualization experts to make it compatible to the virtual environment, such as the aforementioned HIVE Cylinder display. In this work, an integrated pipeline is introduced, promoting direct interaction on semantically-linked 2D and 3D heterogeneous data, which is immersively displayed in a virtual working place. The focus here is on the interactive visualization of heterogeneous molecular data which is combined based on existing or inferred links following hierarchical concept definitions. The user is supported by the developed system with adaptive analyzes customized towards the required tasks. The developed ontology describes both structural biology as well as interaction concepts .
5 Conclusions and Outlook
Here, six new applications along the path from Virtual Reality to Immersive Analytics in the field of Bioinformatics are presented. The introduced work discusses:
- the evaluation of the performance of Augmented Reality-related technology in the field of molecular visualizations,
- aspects of content creation for stereoscopic cell visualization environments,
- interactive modeling of extracellular matrix structures in a virtual environment,
- a new molecular dynamics simulation visualization tool, enabling the collaborative evaluation of underlying structures,
- a new system to immersively explore multiple redox proteomic datasets, and finally
- semantics to describe semantically linked multi-dimensional molecular data and its interactive visualization.
With this overview it is our hope to encourage more Bioinformatics-related research in the area of Immersive Analytics in the future. During the demo session at the SD&A 2018 conference, where all of these approaches were presented, the Head-mounted displays were clearly the winning devices. Their big advantage of being mobile, combining a graphics-driven laptop with an HMD, using the HoloLens, or just by using a smartphone in combination with a Google VR cardboard HMD, enables researchers already today to use VR-related technologies everywhere. In case future approaches can be used in an easily comprehensible and reasonable way, they can fully benefit of their high potential.
Sutherland IE. The ultimate display. Multimedia: From Wagner to virtual reality. 1965. p. 506–8.
Sutherland IE. A head-mounted three dimensional display. In: Proceedings of the December 9–11, 1968, fall joint computer conference, part I. ACM, 1968:757–64.
Heilig ML. Sensorama simulator. 1962. US Patent 3,050,870, Google Patents.
Brooks Jr FP, Ouh-Young M, Batter JJ, Jerome Kilpatrick P. Project GROPEHaptic displays for scientific visualization. In: ACM SIGGraph computer graphics. vol. 24. ACM, 1990:177–85.
Cruz-Neira C, Sandin DJ, DeFanti TA, Kenyon RV, Hart JC. The CAVE: audio visual experience automatic virtual environment. Commun ACM. 1992;35:64–72.
Febretti A, Nishimoto A, Thigpen T, Talandis J, Long L, Pirtle JD, et al. CAVE2: a hybrid reality environment for immersive simulation and information analysis. In: IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 2013:864903–864903–12.
Goscinski WJ, McIntosh P, Felzmann U, Maksimenko A, Hall CJ, Gureyev T, et al. The multi-modal Australian ScienceS Imaging and Visualization Environment (MASSIVE) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Front Neuroinform. 2014;8:30.
Wurtele ES, Bassham DC, Dickerson J, Kabala DJ, Schneller W, Stenerson M, et al. Meta!Blast: a serious game to explore the complexities of structural and metabolic cell biology. In: Proceedings of the ASME 2010 World Conference on Innovative Virtual Reality. ASME, 2010:237–40.
Bryson S. Virtual reality in scientific visualization. Commun ACM. 1996;39:62–71.
Brooks FP. Impressions by a dinosaur – summary of Faraday discussion 169: molecular simulations and visualization. Faraday Discuss. 2014;169:521–7.
Cruz-Neira C, Langley R, Bash PA. VIBE: a virtual biomolecular environment for interactive molecular modeling. Comput Chem. 1996;20:469–77.
Cook KA, Thomas JJ. Illuminating the path: the research and development agenda for visual analytics. Richland, WA (US): Pacific Northwest National Laboratory (PNNL), 2005.
Chandler T, Cordeil M, Czauderna T, Dwyer T, Glowacki J, Goncu C, et al. Immersive analytics. In: Big Data Visual Analytics (BDVA). IEEE, 2015:1–8.
Sommer B, Hamacher A, Kaluza O, Czauderna T, Klapperstück M, Biere N, et al. Stereoscopic space map – semi-immersive configuration of 3D-stereoscopic tours in multi-display environments. In: Electronic Imaging, Proceedings of Stereoscopic Displays and Applications XXVII. 2016;2016:1–9.
Yang Y, Wurtele ES, Cruz-Neira C, Dickerson JA. Hierarchical visualization of metabolic networks using virtual reality. In: Proceedings of the 2006 ACM international conference on virtual reality continuum and its applications. ACM, 2006:377–81.
Sommer B, Wang SJ, Xu L, Chen M, Schreiber F. Hybrid-dimensional visualization and interaction-integrating 2D and 3D visualization with semi-immersive navigation techniques. In: Big Data Visual Analytics (BDVA). IEEE, 2015:1–8.
Sabir K, Stolte C, Tabor B, O’Donoghue S. The Molecular Control Toolkit: controlling 3D molecular graphics via gesture and voice. In: 2013 IEEE Symposium on Biological Data Visualization (BioVis). IEEE, 2013:49–56.
Müller C, Huber M, Biener V, Herr D, Koch S, Reina G, et al. Interactive molecular graphics for augmented reality using HoloLens. J Integr Bioinform. 2018;15. Article ID: 20180005.
Biere N, Ghaffar M, Doebbe A, Jäger D, Rothe N, Friedrich BM, et al. Heuristic modeling and 3D stereoscopic visualization of a Chlamydomonas reinhardtii cell. J Integr Bioinform. 2018;15. Article ID: 20180003.
Hess B, Kutzner C, van der Spoel D, Lindahl E. Gromacs 4: algorithms for highly efficient, load-balanced, and scalable molecular simulation. J Chem Theory Comput. 2008;4:435–47.
Mancera R, Wiebrands M, Malajczuk C, Woods A, Rohl A. Molecular dynamics visualization (MDV): stereoscopic 3D display of biomolecular structure and interactions using the Unity game engine. J Integr Bioinform. 2018;15. Article ID: 20180010.
Belloy N, Wong H, révoteau Jonquet J, Baud S, Dauchez M. Mesoscopic rigid body modelling of the extracellular matrix self-assembly. J Integr Bioinform. 2018;15. Article ID: 20180009.
Doutreligne S, Cragnolini T, Pasquali S, Derreumaux P, Baaden M. UnityMol: interactive scientific visualization for integrative biology. In: Large Data Analysis and Visualization (LDAV), 2014 IEEE 4th Symposium on. IEEE, 2014:109–10.
Maes A, Martinez X, Druart K, Laurent B, Guégan S, Marchand CH, et al. MinOmics, an integrative and immersive tool for multi-omics analysis. J Integr Bioinform. 2018;15. Article ID: 20180006.
Trellet M, Férey N, Flotyński J, Baaden M, Bourdot P. Semantics for an integrative and immersive pipeline combining visualization and analysis of molecular data. J Integr Bioinform. 2018;15. Article ID: 20180004.