Skip to content
Licensed Unlicensed Requires Authentication Published by Oldenbourg Wissenschaftsverlag April 22, 2021

The SmARtphone Controller

Leveraging Smartphones as Input and Output Modality for Improved Interaction within Mobile Augmented Reality Environments

Pascal Knierim ORCID logo, Dimitri Hein, Albrecht Schmidt and Thomas Kosch
From the journal i-com

Abstract

Current interaction modalities for mobile Augmented Reality (AR) are tedious and lack expressiveness. To overcome these prevalent limitations, we developed and evaluated a multimodal interaction concept by pairing a smartphone as an input and output modality for mobile AR. In a user study (n = 24), we investigated the effects on interaction speed, accuracy, and task load for (1) virtual object manipulation as well as (2) interaction with established graphical user interfaces (GUIs). Our findings show that a smartphone-based AR controller results in significantly faster and more accurate object manipulation with reduced task load than state-of-art mid-air gestures. Our results also indicate a significant enhancement for using the physical touchscreen as a modality compared to mid-air gestures for GUI interaction. We conclude that interaction in mobile AR environments can be improved by utilizing a smartphone as an omnipresent controller. Additionally, we discuss how future AR systems can benefit from tangible touchscreens as an additional and complementary interaction modality.

Funding source: Bundesministerium für Bildung und Forschung

Award Identifier / Grant number: 16SV7527 Be-greifen

Funding source: European Research Council

Award Identifier / Grant number: 683008. AMPLIFY

Funding statement: This work was supported by the German Federal Ministry of Education and Research under grant no. 16SV7527 Be-greifen and the European Union’s Horizon 2020 Programme under ERCEA grant no. 683008. AMPLIFY.

References

[1] Mohammed Al-Sada, Fumiko Ishizawa, Junichi Tsurukawa, and Tatsuo Nakajima. 2016. Input Forager: A User-Driven Interaction Adaptation Approach for Head Worn Displays. In Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia (Rovaniemi, Finland) (MUM ’16). Association for Computing Machinery, New York, NY, USA, 115–122. https://doi.org/10.1145/3012709.3012719.10.1145/3012709.3012719Search in Google Scholar

[2] M. S. Arefin, N. Phillips, A. Plopski, J. L. Gabbard, and J. E. Swan. 2020. Impact of AR Display Context Switching and Focal Distance Switching on Human Performance: Replication on an AR Haploscope. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 571–572.10.1109/VRW50115.2020.00137Search in Google Scholar

[3] Teo Babic, Harald Reiterer, and Michael Haller. 2018. Pocket6: A 6DoF Controller Based On A Simple Smartphone Application. In Proceedings of the Symposium on Spatial User Interaction (Berlin, Germany) (SUI ’18), Association for Computing Machinery, New York, NY, USA, 2–10. https://doi.org/10.1145/3267782.3267785.10.1145/3267782.3267785Search in Google Scholar

[4] Huidong Bai, Gun A. Lee, and Mark Billinghurst. 2012. Freeze View Touch and Finger Gesture Based Interaction Methods for Handheld Augmented Reality Interfaces. In Proceedings of the 27th Conference on Image and Vision Computing New Zealand (Dunedin, New Zealand) (IVCNZ ’12). Association for Computing Machinery, New York, NY, USA, 126–131. https://doi.org/10.1145/2425836.2425864.10.1145/2425836.2425864Search in Google Scholar

[5] Rafael Ballagas, Jan O. Borchers, Michael Rohs, and Jennifer G. Sheridan. The Smart Phone – A Ubiquitous Input Device. IEEE Pervasive Computing (2006).10.1109/MPRV.2006.18Search in Google Scholar

[6] Sebastian Boring, Marko Jurmu, and Andreas Butz. 2009. Scroll, Tilt or Move It: Using Mobile Phones to Continuously Control Pointers on Large Public Displays. In Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7, (Melbourne, Australia) (OZCHI ’09). Association for Computing Machinery, New York, NY, USA, 161–168. https://doi.org/10.1145/1738826.1738853.10.1145/1738826.1738853Search in Google Scholar

[7] Rahul Budhiraja, Gun A. Lee, and Mark Billinghurst. 2013. Interaction techniques for HMD-HHD hybrid AR systems. In ISMAR. 243–244.10.1109/ISMAR.2013.6671786Search in Google Scholar

[8] Rahul Budhiraja, Gun A. Lee, and Mark Billinghurst. 2013. Using a HHD with a HMD for mobile AR interaction. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 1–6.10.1109/ISMAR.2013.6671837Search in Google Scholar

[9] Aryzon Cardboard. 2019. Aryzon: Augumented Reality powered by your smartphone. https://www.aryzon.com.Search in Google Scholar

[10] Gerhard Reitmayr, Chris Chiu, Er Kusternig, Michael Kusternig, and Hannes Witzmann. [n. d.] iOrb – Unifying Command and 3D Input for Mobile Augmented Reality. In Proc. IEEE VR Workshop on New Directions in 3D User Interfaces. 7–10.Search in Google Scholar

[11] David Dobbelstein, Christian Winkler, Gabriel Haas, and Enrico Rukzio. PocketThumb: A Wearable Dual-Sided Touch Interface for Cursor-Based Control of Smart-Eyewear. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 2, Article 9 (June 2017) 17 pages. https://doi.org/10.1145/3090055.10.1145/3123024.3123186Search in Google Scholar

[12] David Drascic and Paul Milgram. 1996. Perceptual issues in augmented reality. In Stereoscopic displays and virtual reality systems III, Vol. 2653. International Society for Optics and Photonics, 123–134.10.1117/12.237425Search in Google Scholar

[13] Anna Eiberger, Per Ola Kristensson, Susanne Mayr, Matthias Kranz, and Jens Grubert. 2019. Effects of Depth Layer Switching between an Optical See-Through Head-Mounted Display and a Body-Proximate Display. In Symposium on Spatial User Interaction (New Orleans, LA, USA) (SUI ’19). Association for Computing Machinery, New York, NY, USA, Article 15, 9 pages. https://doi.org/10.1145/3357251.3357588.10.1145/3357251.3357588Search in Google Scholar

[14] Markus Funk, Andreas Bächler, Liane Bächler, Thomas Kosch, Thomas Heidenreich, and Albrecht Schmidt. 2017. Working with Augmented Reality? A Long-Term Analysis of In-Situ Instructions at the Assembly Workplace. In Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments (Island of Rhodes, Greece) (PETRA ’17). Association for Computing Machinery, New York, NY, USA, 222–229. https://doi.org/10.1145/3056540.3056548.10.1145/3056540.3056548Search in Google Scholar

[15] Markus Funk, Thomas Kosch, and Albrecht Schmidt. 2016. Interactive Worker Assistance: Comparing the Effects of Head-Mounted Displays, In-Situ Projection, Tablet, and Paper Instructions. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. https://doi.org/10.1145/2971648.2971706.10.1145/2971648.2971706Search in Google Scholar

[16] J. L. Gabbard, D. G. Mehra, and J. E. Swan. Effects of AR Display Context Switching and Focal Distance Switching on Human Performance. IEEE Transactions on Visualization and Computer Graphics 25, 6 (June 2019), 2228–2241. https://doi.org/10.1109/TVCG.2018.2832633.10.1109/TVCG.2018.2832633Search in Google Scholar PubMed

[17] Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015. MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3933–3942. https://doi.org/10.1145/2702123.2702331.10.1145/2702123.2702331Search in Google Scholar

[18] Taejin Ha and Woontack Woo. 2011. ARWand: Phone-Based 3D Object Manipulation in Augmented Reality Environment. In 2011 International Symposium on Ubiquitous Virtual Reality (ISUVR). IEEE, 44–47.10.1109/ISUVR.2011.14Search in Google Scholar

[19] Sandra G. Hart. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9 (Nov. 2016), 904–908.10.1037/e577632012-009Search in Google Scholar

[20] Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1063–1072. https://doi.org/10.1145/2556288.2557130.10.1145/2556288.2557130Search in Google Scholar

[21] Juan David Hincapié-Ramos, Kasim Ozacar, Pourang P. Irani, and Yoshifumi Kitamura. 2015. GyroWand: IMU-Based Raycasting for Augmented Reality Head-Mounted Displays. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction (Los Angeles, California, USA) (SUI ’15). Association for Computing Machinery, New York, NY, USA, 89–98. https://doi.org/10.1145/2788940.2788947.10.1145/2788940.2788947Search in Google Scholar

[22] Teresa Hirzle, Jan Rixen, Jan Gugenheimer, and Enrico Rukzio. 2018. WatchVR: Exploring the Usage of a Smartwatch for Interaction in Mobile Virtual Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188629.10.1145/3170427.3188629Search in Google Scholar

[23] Jinki Jung, Jihye Hong, Sungheon Park, and Hyun S. Yang. 2012. Smartphone as an Augmented Reality Authoring Tool via Multi-Touch Based 3D Interaction Method. In Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (Singapore, Singapore) (VRCAI ’12). Association for Computing Machinery, New York, NY, USA, 17–20. https://doi.org/10.1145/2407516.2407520.10.1145/2407516.2407520Search in Google Scholar

[24] Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: Smartwatch-Based 3D Pointing for Smartphone-Based Virtual Reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST ’16). Association for Computing Machinery, New York, NY, USA, 365–366. https://doi.org/10.1145/2993369.2996311.10.1145/2993369.2996311Search in Google Scholar

[25] Konstantin Klamka and Raimund Dachselt. 2018. ARCord: Visually Augmented Interactive Cords for Mobile Interaction. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188456.10.1145/3170427.3188456Search in Google Scholar

[26] Jingbo Liu, Oscar Kin-Chung Au, Hongbo Fu, and Chiew-Lan Tai. Two-Finger Gestures for 6DOF Manipulation of 3D Objects. Computer Graphics Forum 31, 7 (2012), 2047–2055. https://doi.org/10.1111/j.1467-8659.2012.03197.x. arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2012.03197.x.10.1111/j.1467-8659.2012.03197.xSearch in Google Scholar

[27] Tariq Masood and Johannes Egger. Augmented reality in support of Industry 4.0 – Implementation challenges and success factors. Robotics and Computer-Integrated Manufacturing 58 (2019), 181–195.10.1016/j.rcim.2019.02.003Search in Google Scholar

[28] Sven Mayer, Lars Lischke, Jens Emil Grønbæk, Zhanna Sarsenbayeva, Jonas Vogelsang, Paweł W. Woundefinedniak, Niels Henze, and Giulio Jacucci. 2018. Pac-Many: Movement Behavior When Playing Collaborative and Competitive Games on Large Displays. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3173574.3174113.10.1145/3173574.3174113Search in Google Scholar

[29] Mark Mine, Arun Yoganandan, and Dane Coffey. 2014. Making VR Work: Building a Real-World Immersive Modeling Application in the Virtual World. In Proceedings of the 2nd ACM Symposium on Spatial User Interaction (Honolulu, Hawaii, USA) (SUI ’14). Association for Computing Machinery, New York, NY, USA, 80–89. https://doi.org/10.1145/2659766.2659780.10.1145/2659766.2659780Search in Google Scholar

[30] Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815.10.1145/3290605.3300815Search in Google Scholar

[31] Florian Müller, Joshua McManus, Sebastian Günther, Martin Schmitz, Max Mühlhäuser, and Markus Funk. 2019. Mind the Tap: Assessing Foot-Taps for Interacting with Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300707.10.1145/3290605.3300707Search in Google Scholar

[32] E. Normand and M. J. McGuffin. 2018. Enlarging a Smartphone with AR to Create a Handheld VESAD (Virtually Extended Screen-Aligned Display). In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 123–133. https://doi.org/10.1109/ISMAR.2018.00043.10.1109/ISMAR.2018.00043Search in Google Scholar

[33] Krzysztof Pietroszek, James R. Wallace, and Edward Lank. 2015. Tiltcasting: 3D Interaction on Large Displays Using a Mobile Device. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 57–62. https://doi.org/10.1145/2807442.2807471.10.1145/2807442.2807471Search in Google Scholar

[34] Umar Rashid, Miguel A. Nacenta, and Aaron Quigley. 2012. The Cost of Display Switching: A Comparison of Mobile, Large Display and Hybrid UI Configurations. In Proceedings of the International Working Conference on Advanced Visual Interfaces (Capri Island, Italy) (AVI ’12). Association for Computing Machinery, New York, NY, USA, 99–106. https://doi.org/10.1145/2254556.2254577.10.1145/2254556.2254577Search in Google Scholar

[35] Houssem Saidi, Marcos Serrano, Pourang Irani, and Emmanuel Dubois. 2017. TDome: A Touch-Enabled 6DOF Interactive Device for Multi-Display Environments. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 5892–5904. https://doi.org/10.1145/3025453.3025661.10.1145/3025453.3025661Search in Google Scholar

[36] Stefan Schneegass and Alexandra Voit. 2016. GestureSleeve: Using Touch Sensitive Fabrics for Gestural Input on the Forearm for Controlling Smartwatches. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (Heidelberg, Germany) (ISWC ’16). Association for Computing Machinery, New York, NY, USA, 108–115. https://doi.org/10.1145/2971763.2971797.10.1145/2971763.2971797Search in Google Scholar

[37] Marcos Serrano, Dale Hildebrandt, Sriram Subramanian, and Pourang Irani. 2014. Identifying Suitable Projection Parameters and Display Configurations for Mobile True-3D Displays. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (Toronto, ON, Canada) (MobileHCI ’14). Association for Computing Machinery, New York, NY, USA, 135–143. https://doi.org/10.1145/2628363.2628375.10.1145/2628363.2628375Search in Google Scholar

[38] Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI ’19). Association for Computing Machinery, New York, NY, USA, Article 13, 13 pages. https://doi.org/10.1145/3290605.3300243.10.1145/3290605.3300243Search in Google Scholar

[39] Philipp Wacker, Oliver Nowak, Simon Voelker, and Jan Borchers. 2019. ARPen: Mid-Air Object Manipulation Techniques for a Bimanual AR System with Pen & Smartphone. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300849.10.1145/3290605.3300849Search in Google Scholar

[40] W. Zhao and V. Madhavan. 2005. Integration of voice commands into a virtual reality environment for assembly design. In Proceedings of the 10th annual international conference on industrial engineering theory, applications & practice (Clearwater Beach, FL, USA).Search in Google Scholar

[41] Xianjun Sam Zheng, Cedric Foucault, Patrik Matos da Silva, Siddharth Dasari, Tao Yang, and Stuart Goose. 2015. Eye-Wearable Technology for Machine Maintenance: Effects of Display Position and Hands-Free Operation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 2125–2134. https://doi.org/10.1145/2702123.2702305.10.1145/2702123.2702305Search in Google Scholar

[42] Fengyuan Zhu and Tovi Grossman. 2020. BISHARE: Exploring Bidirectional Interactions Between Smartphones and Head-Mounted Augmented Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376233.10.1145/3313831.3376233Search in Google Scholar

Published Online: 2021-04-22
Published in Print: 2021-04-27

© 2021 Walter de Gruyter GmbH, Berlin/Boston