Jump to ContentJump to Main Navigation
Show Summary Details
More options …


Journal of Interactive Media

Editor-in-Chief: Ziegler, Jürgen

3 Issues per year

See all formats and pricing
More options …
Volume 14, Issue 2


Hybrid Avatar-Agent Technology – A Conceptual Step Towards Mediated “Social” Virtual Reality and its Respective Challenges

Daniel Roth
  • Corresponding author
  • Media and Communication Psychology, Department of Psychology, University of Cologne, Cologne, Germany; and Human-Computer Interaction, Institute for Computer Science, University of Würzburg, Würzburg, Germany
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Prof. Dr. Marc Erich Latoschik / Prof. Dr. Dr. Kai Vogeley
  • Cognitive Neuroscience, Forschungszentrum Jülich, Jülich, Germany; and Department of Psychiatry, University Hospital of Cologne, Cologne, Germany
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Prof. Dr. Gary Bente
  • Media and Communication Psychology, Department of Psychology, University of Cologne, Cologne, Germany; and Department of Communication, Michigan State University, East Lansing, USA
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
Published Online: 2015-07-12 | DOI: https://doi.org/10.1515/icom-2015-0030


Driven by large industry investments, developments of Virtual Reality (VR) technologies including unobtrusive sensors, actuators and novel display devices are rapidly progressing. Realism and interactivity have been postulated as crucial aspects of immersive VR since the naissance of the concept. However, today’s VR still falls short from creating real life-like experiences in many regards. This holds particularly true when introducing the “social dimension” into the virtual worlds. Apparently, creating convincing virtual selves and virtual others and conveying meaningful and appropriate social behavior still is an open challenge for future VR. This challenge implies both, technical aspects, such as the real-time capacities of the systems, but also psychological aspects, such as the dynamics of human communication. Our knowledge of VR systems is still fragmented with regard to social cognition, although the social dimension is crucial when aiming at autonomous agents with a certain social background intelligence. It can be questioned though whether a perfect copy of real life interactions is a realistic or even meaningful goal of social VR development at this stage. Taking into consideration the specific strengths and weaknesses of humans and machines, we propose a conceptual turn in social VR which focuses on what we call “hybrid avatar-agent systems”. Such systems are required to generate i) avatar mediated interactions between real humans, taking advantage of their social intuitions and flexible communicative skills and ii) an artificial social intelligence (AIS) which monitors, and potentially moderates or transforms ongoing virtual interactions based on social signals, such as performing adaptive manipulations of behavior in intercultural conversations. The current article sketches a respective base architecture and discusses necessary research prospects and challenges as a starting point for future research and development.

Keywords: Computer-Mediated Communication; Human-Computer Interaction; Interpersonal Synchronization; Virtual Reality


  • Andersen, P. A. 2015. In different dimensions: nonverbal communication and culture. In: (L. A. Samovar and R. E. Porter, eds) Intercultural communication: a reader (Fourteenth Edition and Fortieth Anniversary Edition. Cengage Learning, Boston, MA, pp. 229–241.Google Scholar

  • Bailenson, J. N., A. C. Beall, J. Loomis, J. Blascovich and M. Turk. 2004. Transformed social interaction: decoupling representation from behavior and form in collaborative virtual environments. Presence-Teleop. Virt. 13(4): 428–441.Google Scholar

  • Bailenson, J. and J. Blascovich. 2004. Avatars. In: Encyclopedia of Human-Computer Interaction, Berkshire Publishing Group, 64–68.Google Scholar

  • Baker, C. L. and O. J. Braddick. 1985. Eccentricity-dependent scaling of the limits for short-range apparent motion perception. Vis. Res. 25(6): 803–812.Google Scholar

  • Bente, G., W. C. Donaghy, and D. Suwelack. 1998. Sex differences in body movement and visual attention: an integrated analysis of movement and gaze in mixed-sex dyads. J. Nonverbal Behav. 22(1): 31–58.Google Scholar

  • Bente, G., F. Eschenburg and N. C. Krämer. 2007. Virtual gaze. A pilot study on the effects of computer simulated gaze in avatar-based conversations. In: (R. Shumaker, ed) Virtual reality, HCII 2007, LNCS 4563, Springer, 185–194.Google Scholar

  • Bente, G., N. C. Krämer and F. Eschenburg. 2008. Is there anybody out there? In: (E. A. Konijn, S. Utz, M. Tanis and S. B. Barnes, eds) Mediated interpersonal communication, 131–157.Google Scholar

  • Bente, G., H. Leuschner, A. A. Issa and J. J. Blascovich. 2010. The others:universals and cultural specificities in the perception of status and dominance from nonverbal behavior. Conscious. Cogn. 19(3): 762–777.Google Scholar

  • Betella, A., D. Pacheco, R. Zucca, X. D. Arsiwalla, P. Omedas, Lanatà, A. et al. 2014. Interpreting psychophysiological states using unobtrusive wearable sensors in virtual reality. ACHI 2014, The Seventh International Conference on Advances in Computer-Human Interactions, 331–336.Google Scholar

  • Bowman, D. A., E. Kruijff, J. Joseph, J. LaViola and I. Poupyrev. 2005. 3D user interfaces – Theory and practice. Addison-Wesley.Google Scholar

  • Bradley, J., O. Mival and D. Benyon. 2009. Wizard of Oz experiments for companions. Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology, British Computer Society, 313–317.Google Scholar

  • Buttazzo, G. C. 2011. Hard real-time computing systems. Springer, Boston, MA.Google Scholar

  • Buxton, B. 2008. The long nose of innovation. Insight, 11: 27.Google Scholar

  • Chanel, G., Kivikangas, J. M. and N. Ravaja. 2012. Physiological compliance for social gaming analysis: Cooperative versus competitive play. Interact. Comput. 24(4): 306–316.Google Scholar

  • Culnan, M. J. and M. L. Markus. 1987. Information technologies.Google Scholar

  • Dumas, G., J. Nadel, R. Soussignan, J. Martinerie and L. Garnero. 2010. Inter-brain synchronization during social interaction. PLoS ONE, 5(8): e12166.Google Scholar

  • Endrass, B., E. André, M. Rehm and Y. Nakano. 2013. Investigating culture-related aspects of behavior for virtual characters. Auton. Agent Multi Agent Syst. 27(2): 277–304.Google Scholar

  • Frey, S. 1984. Die nonverbale Kommunikation (The nonverbal communication). SEL-Foundation for technical and economical communication science in the society of funding for german science.Google Scholar

  • Gumzej, R. and W. A. Halang. 2010. Real-time systems’ quality of service. Springer, London.Google Scholar

  • Hale, K. S. and K. M. Stanney. 2014. Handbook of virtual environments: Design, implementation, and applications. CRC Press.Google Scholar

  • Izadi, S., D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, et al. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ACM, pp. 559–568.Google Scholar

  • Katz, J. E. 2014. Living inside mobile social information. Greyden Press, Dayton, OH.Google Scholar

  • Krämer, K. 2010. Real or fake – how does motor mimicry influence the perception of human interactions? (Diploma thesis). University of Cologne, Cologne.Google Scholar

  • Kuzmanovic, B., A. L. Georgescu, S. B. Eickhoff, N. J. Shah, G. Bente, G. R. Fink and K. Vogeley. 2009. Duration matters: dissociating neural correlates of detection and evaluation of social gaze. Neuroimage, 46(4), 1154–1163.CrossrefGoogle Scholar

  • Lakens, D. and M. Stel. 2011. If they move in sync, they must feel in sync: movement synchrony leads to attributions of rapport and entitativity. Soc. Cognition, 29(1): 1–14.CrossrefGoogle Scholar

  • Lakin, J. L. and T. L. Chartrand. 2003. Using nonconscious behavioral mimicry to create affiliation and rapport. Psychol. Sci. 14(4): 334–339.Google Scholar

  • Latoschik, M. E. 2005. A User Interface Framework for Multimodal VR Interactions. In Proceedings of the IEEE seventh International Conference on Multimodal Interfaces, ICMI 2005, 76–83. Trento, Italy.Google Scholar

  • Latoschik, M. E. 2014a. Lecture notes – real-time interactive systems – Julius Maximilian University Würzburg. Würzburg.Google Scholar

  • Latoschik, M. E. 2014b. Smart Graphics / Intelligent Graphics. Informatik Spektrum, Vol. 37 (1), pp. 36–41. Springer.Google Scholar

  • Latoschik, M. E. and Blach, R. 2008. Semantic modelling for virtual worlds – a novel paradigm for realtime interactive systems? In: Proceedings of the ACM VRST 2008, 17–20.Google Scholar

  • Latoschik, M. E. and M. Fischbach. 2014. Engineering variance: software techniques for scalable, customizable, and reusable multimodal processing. M. Kurosu (Hrsg.), Human Computer Interaction, Part I, HCI 2014 (Vol. 8510, 308–319). Springer International Publishing.Google Scholar

  • Lee, K. M. 2004. Presence, explicated. Commun.Theory. 14(1): 27–50.Google Scholar

  • Lievesley, R., M. Wozencroft and D. Ewins. 2011. The Emotiv EPOC neuroheadset: an inexpensive method of controlling assistive technologies using facial expressions and thoughts? J. Assist. Technol.5(2): 67–82.Google Scholar

  • Lugrin, J.-L., J. Latt and M. E. Latoschik. 2015. Avatar anthropomorphism and illusion of body ownership in VR. Proceedings of the IEEE VR 2015.Google Scholar

  • Matsumoto, D. 2006. Culture and nonverbal behavior. Handbook of nonverbal communication, 219–235.Google Scholar

  • Millán, J., R. Rupp, G. R. Müller-Putz, R. Murray-Smith, C. Giugliemma, M. Tangermann, C. Vidaurre, F. Cicotti, A. Kübler, R. Leeb, C. Neuper, K. R. Müller and D. Mattia. 2010. Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges. Front. Neurosci. 4, 161.Google Scholar

  • Pfeiffer, U. J., L. Schilbach, M. Jording, B. Timmermans, G. Bente and K. Vogeley. 2012. Eyes on the mind: investigating the influence of gaze dynamics on the perception of others in real-time social interaction. Front. Psychol. 3.Google Scholar

  • Pfeiffer, U. J., L. Schilbach, B. Timmermans, B. Kuzmanovic, A. L. Georgescu, G. Bente and K. Vogeley. 2014. Why we interact: on the functional role of the striatum in the subjective experience of social interaction. NeuroImage, 101, 124–137.Google Scholar

  • Pfeiffer, U. J., B. Timmermans, G. Bente, K. Vogeley and L. Schilbach. 2011. A non-verbal turing test: differentiating mind from machine in gaze-based social interaction. PLoS ONE, 6(11), e27591.Google Scholar

  • Rehfeld, S., H. Tramberend and M. E. Latoschik. 2014. Profiling and benchmarking event- and message-passing-based asynchronous Realtime Interactive Systems. ACM VRST 2014 Conference Proceedings.Google Scholar

  • Riek, L. D. 2012. Wizard of Oz studies in HRI: a systematic review and new reporting guidelines. J. Hum.-Rob. Interact. 1(1).Google Scholar

  • Riggio, R. E. 2006. Nonverbal skills and abilities. The Sage handbook of nonverbal communication, 79–96.Google Scholar

  • Rivera, J. and R. van der Meulen. 2014. Gartner’s 2014 Hype Cycle for emerging technologies maps the journey to digital business. Retrieved from http://www.gartner.com/newsroom/id/2819918 (April 2015).

  • Sanders, J. A. and R. L. Wiseman. 1990. The effects of verbal and nonverbal teacher immediacy on perceived cognitive, affective, and behavioral learning in the multicultural classroom. Communic. Educ.39(4): 341–353.Google Scholar

  • Sebanz, N., H. Bekkering and Knoblich, G. (2006). Joint action: bodies and minds moving together. Trends Cogn. Sci. 10(2): 70–76.Google Scholar

  • Slater, M. (2014). Grand Challenges in Virtual Environments. Front. Robot. Artif. Intel. 1(3).Google Scholar

  • Spapé, M. M., J. M. Kivikangas, S. Järvelä, I. Kosunen, G. Jacucci and N. Ravaja. 2013. Keep your opponents close: social context affects EEG and fEMG linkage in a turn-based computer game. PLoS ONE, 8(11): e78795.Google Scholar

  • Thornton, I. M. (1998). The visual perception of human locomotion. Cogn. Neuropsychol. 15(6–8): 535–552.Google Scholar

  • Vinciarelli, A., M. Pantic and H. Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image. Vis. Comput. 27(12): 1743–1759.Google Scholar

  • Wagner, J., F. Lingenfelser, T. Baur, I. Damian, F. Kistler and E. André. 2013. The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. Proceedings of the 21st ACM international conference on Multimedia, ACM, 831–834.Google Scholar

  • Walther, J. B., B. Van Der Heide, A. Ramirez Jr, J. K. Burgoon and J. Peña. 2015. Interpersonal and hyperpersonal dimensions of computer-mediated communication. In: (S. S. Sundar, ed) Handbook of the psychology of communication technology. John Wiley & Sons, Inc.Google Scholar

  • Yan, Y., S. H. Konduri, A. Kulkarni, V. Anand, S. Y. Ko and L. Ziarek. 2013. Rtdroid: a design for real-time android. In: Proceedings of the 11th International Workshop on Java Technologies for Real-time and Embedded Systems, ACM, 98–107.Google Scholar

  • Yee, N. and J. N. Bailenson. 2006. Walk a mile in digital shoes: the impact of embodied perspective-taking on the reduction of negative stereotyping in immersive virtual environments. Proceedings of PRESENCE, 24–26.Google Scholar

About the article

Daniel Roth

Daniel Roth, M. Eng. received his Master’s degree in Media and Imaging Technology from the University of Applied Sciences, Cologne and is currently working as research associate at the Professorship for Media- and Communication Psychology of the University of Cologne. He started his doctoral thesis on interpersonal synchronization in Virtual Realities at the Human-Computer Interaction Group of the University of Würzburg.

Prof. Dr. Marc Erich Latoschik

Marc Erich Latoschik holds the chair for Human-Computer Interaction of the University of Würzburg. His work is interdisciplinary oriented with a strong focus on multimodal interaction and intelligent graphics interconnecting real-time 3D simulations, Virtual and Augmented Reality, Artificial Intelligence, and cognitive sciences.

Prof. Dr. Dr. Kai Vogeley

Kai Vogeley is a Full Professor at the Department of Psychiatry of the University of Cologne and affiliated at the Institute of Neuroscience and Medicine – Cognitive Neuroscience, Research Center Jülich, where he leads the Social Cognition Group. His research focuses on social neuroscience with a special focus on nonverbal communication and interaction. Clinically, his main interest is autism spectrum disorder in adulthood.

Prof. Dr. Gary Bente

Gary Bente is a psychologist and Full Professor at the Department of Psychology of the University of Cologne and head of the Unit for Media and Communication Psychology. He is appointed as a Visiting Professor at the Department of Communication at Michigan State University, USA. His research focuses on the analysis of movement and gaze behavior in interpersonal communication including face-to-face as well as mediated interactions.

Published Online: 2015-07-12

Published in Print: 2015-08-01

Citation Information: i-com, Volume 14, Issue 2, Pages 107–114, ISSN (Online) 2196-6826, ISSN (Print) 1618-162X, DOI: https://doi.org/10.1515/icom-2015-0030.

Export Citation

© 2015 Walter de Gruyter GmbH, Berlin/Boston. Copyright Clearance Center

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Kai Vogeley
Philosophical Transactions of the Royal Society B: Biological Sciences, 2017, Volume 372, Number 1727, Page 20160245
Klaus Hesse, Philipp A. Schroeder, Jonathan Scheeff, Stefan Klingberg, and Christian Plewnia
Journal of Behavior Therapy and Experimental Psychiatry, 2017, Volume 56, Page 129

Comments (0)

Please log in or register to comment.
Log in