Benny Liebold is a researcher in the Institute for Media Research at Chemnitz University of Technology. His research focusses on the cognitive and emotional processing of virtual environments with an emphasis on the role of emotions in HCI, presence, game studies, and media effects in general, such as skill transfer and aggressive behavior.
Dipl. Inf. René Richter is a researcher in the department of computer science at Chemnitz University of Technology. His research field is computational neuroscience, in particular the influence of emotions on attention and how this interaction can be used in the context of HCI.
Dipl. Inf. Michael Teichmann works as a researcher in the department of computer science at Chemnitz University of Technology. His research focusses on neuroscientifically grounded computational models of the human visual cortex, in particular self-organization via neural plasticity in recurrent models of the ventral visual stream.
Fred H. Hamker is a professor of artificial intelligence in the department of computer science at Chemnitz University of Technology since 2009. He received his diploma in electrical engineering from the University of Paderborn in 1994 and his Ph. D. in computer science at Ilmenau University of Technology in 1999. He was a postdoc at the Goethe University Frankfurt and the California Institute of Technology (Pasadena, USA). In 2008 he received his venia legendi from the Department of Psychology at the University of Münster.
Peter Ohler is a professor of media psychology at the Institute for Media Research at Chemnitz University of Technology since 2002. He studied psychology at Saarland University and received his Ph. D. at TU Berlin in 1991. He had postdoc positions at TU Berlin and the University of Passau. In 2000 he received his venia legendi in psychology from TU Berlin. His research interests include the psychology of film, evolutionary psychology, cognitive science, and the psychology of play.
Current models for automated emotion recognition are developed under the assumption that emotion expressions are distinct expression patterns for basic emotions. Thereby, these approaches fail to account for the emotional processes underlying emotion expressions. We review the literature on human emotion processing and suggest an alternative approach to affective computing. We postulate that the generalizability and robustness of these models can be greatly increased by three major steps: (1) modeling emotional processes as a necessary foundation of emotion recognition; (2) basing models of emotional processes on our knowledge about the human brain; (3) conceptualizing emotions based on appraisal processes and thus regarding emotion expressions as expressive behavior linked to these appraisals rather than fixed neuro-motor patterns. Since modeling emotional processes after neurobiological processes can be considered a long-term effort, we suggest that researchers should focus on early appraisals, which evaluate intrinsic stimulus properties with little higher cortical involvement. With this goal in mind, we focus on the amygdala and its neural connectivity pattern as a promising structure for early emotional processing. We derive a model for the amygdala-visual cortex circuit from the current state of neuroscientific research. This model is capable of conditioning visual stimuli with body reactions to enable rapid emotional processing of stimuli consistent with early stages of psychological appraisal theories. Additionally, amygdala activity can feed back to visual areas to modulate attention allocation according to the emotional relevance of a stimulus. The implications of the model considering other approaches to automated emotion recognition are discussed.
Barlett, M. S., G. Littlewort, P. Braathen, T. J. Sejnowski and J. R. Movellan. 2003. A prototype for automatic recognition of spontaneous facial actions. In: (S. Thrun and L. K. S. Saul, B., eds.), Advances in Neural Information Processing Systems. NIPS, Vancouver, CA, pp. 1271–1278.
Bartlett, M. S., G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan. 2006. Fully automatic facial action recognition in spontaneous behavior. 7th International Conference on Automatic Face and Gesture Recognition (FGR06).
Becker-Asano, C. 2013. WASABI for affect simulation in human-computer interaction: architecture description and example applications. Paper presented at the Emotion Representations and Modelling for HCI Systems Workshop, Sydney, Australia.
Becker-Asano, C. and I. Wachsmuth. 2009. Affective computing with primary and secondary emotions in a virtual human. Auton. Agent Multi Agent Syst. 20(1): 32–49. doi:
Blascovich, J. 2002. A theoretical model of social influence for increasing the utility of collaborative virtual environments. In: (W. Broll, C. Greenhalgh and E. F. Churchill, eds), Collaborative virtual environments: Proceedings of the 4th international conference on collaborative virtual environments ACM, New York, pp. 25–30.
Bonda, E. 2000. Organization of connections of the basal and accessory basal nuclei in the monkey amygdala. Eur. J. Neurosci. 12(6): 1971–1992.
Bosse, T. and E. Zwanenburg. 2009. There‘s always hope: enhancing agent believability through expectation-based emotions. 1–8. doi:
Cassell, J. and K. R. Thorisson. 1999. The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents. Appl. Artif. Intell. 13(4–5): 519–538. doi: 10.1080/088395199117360)| false
Catani, M., D. K. Jones, R. Donato and D. H. Ffytche. 2003). Occipito-temporal connections in the human brain. Brain. 126(9): 2093–2107.
Creem, S. H. and D. R. Proffitt. 2001. Defining the cortical visual systems: “What”, “Where”, and “How”. Acta Psychol. 107(1–3): 43–68.
de Melo, C. M., P. Carnevale and J. Gratch. (2012). The effect of virtual agents’ emotion displays and appraisals on people’s decision making in negotiation. In: (Y. Nakano, M. Neff, A. Paiva and M. Walker, eds) Intelligent virtual agents, 12th International Conference, IVA 2012, Santa Cruz, CA, USA. Springer, New York, pp. 53–66.
de Melo, C. M., P. J. Carnevale, S. J. Read and J. Gratch. 2014. Reading people‘s minds from emotion expressions in interdependent decision making. J. Pers. Soc. Psychol. 106(1): 73–88. doi:
de Melo, C. M., P. J. Carnevale, S. J. Read and J. Gratch. 2014. Reading people‘s minds from emotion expressions in interdependent decision making. J. Pers. Soc. Psychol. 106(1): 73–88. doi: 10.1037/a0034251)| false
Demeure, V., R. Niewiadomski and C. Pelachaud. 2011. How is believability of a virtual agent related to warmth, competence, personification, and embodiment? Presence: teleoperators and virtual environments. 20(5): 431–448. doi:
Demeure, V., R. Niewiadomski and C. Pelachaud. 2011. How is believability of a virtual agent related to warmth, competence, personification, and embodiment? Presence: teleoperators and virtual environments. 20(5): 431–448. doi: 10.1162/PRES_a_00065)| false
Ekman, P. 1972. Universals and cultural differences in facial expression of emotion. In: (J. D. Cole, ed) Nebraska symposium on motivation, 1971. University of Nebraska Press, Lincoln, pp. 207–282.
Fischer, A. H. and A. S. R. Manstead. 2008. Social functions of emotion. In: (M. Lewis, J. Haviland-Jones and L. Feldmann-Barrett, eds) Handbook of emotions, 3rd ed. Guilford Press, New York, pp. 456–468.
Frijda, N. H. 1986. The emotions. Studies in emotion and social interaction. Cambridge University Press, Cambridge, MA.
Gratch, J. and S. Marsella. 2004. A domain-independent framework for modeling emotion. Cogn. Syst. Res. 5(4): 269–306.
Gratch, J. and S. Marsella. 2005. Lessons from emotion psychology for the design of lifelike characters. Appl. Artif. Intell. 19(3–4): 215–233. doi:
Guadagno, R. E., J. Blascovich, J. N. Bailenson and C. McCall. 2007. Virtual humans and persuasion: the effects of agency and behavioral realism. Media Psychol. 10(1): 1–22. doi: 10.108/15213260701300865)| false
James, W. 1884. What is an emotion? Mind. 9(34): 188–205.
Javier, G., D. Sundgren, R. Rahmani, A. Larsson, A. Moran and I. Bonet. 2015. Speech emotion recognition in emotional feedback for Human-Robot Interaction. IJARAI. 4(2).
Kane, M. J. and R. W. Engle. 2002. The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: an individual-differences perspective. Psychon. Bull. Rev. 9(4): 637–671. doi:
Kane, M. J. and R. W. Engle. 2002. The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: an individual-differences perspective. Psychon. Bull. Rev. 9(4): 637–671. doi: 10.3758/bf03196323)| false
Kleinsmith, A. and N. Bianchi-Berthouze. 2013. Affective body expression perception and recognition: a survey. IEEE Trans. Affect. Comput. 4(1): 15–33.
Koelstra, S., M. Pantic and I. Patras. 2010. A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE J. PAMI. 32(11): 1940–1954.
Kotsia, I., S. Zafeiriou, N. Nikolaidis and I. Pitas. 2008. Texture and shape information fusion for facial action unit recognition. First International Conference on Advances in Computer-Human Interaction.
Krämer, N. C., I. A. Iurgel and G. Bente. 2005. Emotion and motivation in embodied conversational agents. Paper presented at the AISB‘05 Convention, Symposium on Agents that Want and Like: Motivational and Emotional Roots of Cognition and Action, Hatfield, UK.
Krämer, N. C., S. Kopp, C. Becker-Asano and N. Sommer. 2013. Smile and the world will smile with you — the effects of a virtual agent‘s smile on users’ evaluation and behavior. Int. J. Hum. Comput. Stud. 71(3): 335–349. doi:
Krämer, N. C., S. Kopp, C. Becker-Asano and N. Sommer. 2013. Smile and the world will smile with you — the effects of a virtual agent‘s smile on users’ evaluation and behavior. Int. J. Hum. Comput. Stud. 71(3): 335–349. doi: 10.1016/j.ijhcs.2012.09.006)| false
Kukla, E. and P. Nowak. 2015. Facial emotion recognition based on cascade of neural networks. Adv. Intel. Syst. Comput. 67–78.
Lahbiri, M., A. Fnaiech, M. Bouchouicha, M. Sayadi and P. Gorce. 2013. Facial emotion recognition with the hidden Markov model. 2013 International Conference on Electrical Engineering and Software Applications.
Lazarus, R. S. 1991. Emotion and adaption. Oxford University Press,Oxford, UK.
LeDoux, J. E., J. Iwata, P. Cicchetti and D. J. Reis. 1988. Different projections of the central amygdaloid nucleus mediate autonomic and behavioral correlates of conditioned fear. J. Neurosci. 8(7): 2517–2529.
Lee, J. and S. C. Marsella. 2010. Predicting speaker head nods and the effects of affective information. IEEE Trans. Multimedia. 12(6): 552–562. doi:
Li, Y., S. Wang, Y. Zhao and Q. Ji. 2013. Simultaneous facial feature tracking and facial expression recognition. IEEE Trans. Image Process. 22(7): 2559–2573.
Liebold, B. and P. Ohler. 2013. Multimodal emotion expressions of virtual agents. Mimic and vocal emotion expressions and their effects on emotion recognition. In: (T. Pun, C. Pelachaud and N. Sebe, eds) 2013 Humaine Association conference on Affective Computing and Intelligent Interaction, ACII 2013. IEEE, Los Alamitos, CA, pp. 405–410.
Lin, H.-C., S.-C. Mao, C.-L. Su and P.-W. Gean. 2010. Alterations of excitatory transmission in the lateral amygdala during expression and extinction of fear memory. Int. J. Neuropsychopharmacol. 13(3): 335–345.
Lindquist, K. A., T. D. Wager, H. Kober, E. Bliss-Moreau and L. F. Barrett. 2012. The brain basis of emotion: a meta-analytic review. Behav. Brain Sci. 35(3): 121–143. doi:
Lindquist, K. A., T. D. Wager, H. Kober, E. Bliss-Moreau and L. F. Barrett. 2012. The brain basis of emotion: a meta-analytic review. Behav. Brain Sci. 35(3): 121–143. doi: 10.1017/S0140525X11000446)| false
Lozano-Monasor, E., M. T. López, A. Fernández-Caballero and F. Vigo-Bustos. 2014. Facial expression recognition from webcam based on active shape models and support vector machines. Lect. Notes Comput. Sc. 147–154.
Manstead, A. S. R. and A. H. Fischer. 2001. Social appraisal: the social world as object of and influence on appraisal processes. In: (K. R. Scherer, A. Schorr and T. Johnstone, eds) Appraisal processes in emotion: theory, research, application. Oxford University Press, New York, pp. 221–232.
McDonald, A. J. 1998. Cortical pathways to the mammalian amygdala. Prog. Neurobiol. 55(3): 257–332.
Mehrabian, A. and J. A. Russell. 1974. An approach to environmental psychology. MIT Press, Cambridge, MA.
Mulligan, K. and K. R. Scherer. 2012. Toward a working definition of emotion. Emot. Rev. 4(4): 345–357. doi:
Oatley, K. and J. M. Jenkins. 1996. Understanding emotions. Blackwell, Oxford, UK.
Okon-Singer, H., T. Hendler, L. Pessoa and A. J. Shackman. 2015. The neurobiology of emotion-cognition interactions: fundamental questions and strategies for future research. Front. Hum. Neurosci. 9: 58. doi:
Okon-Singer, H., T. Hendler, L. Pessoa and A. J. Shackman. 2015. The neurobiology of emotion-cognition interactions: fundamental questions and strategies for future research. Front. Hum. Neurosci. 9: 58. doi: 10.3389/fnhum.2015.00058)| false
Pessoa, L. and R. Adolphs. 2010. Emotion processing and the amygdala: from a ‘low road’ to ‘many roads’ of evaluating biological significance. Nat. Rev. Neurosci. 11(11): 773–783. doi: 10.1038/nrn2920)| false
Phan, K. L., T. Wager, S. F. Taylor and I. Liberzon. 2002. Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. Neuroimage. 16(2): 331–348. doi: 10.1006/nimg.2002.1087)| false
Phillips, A. T., H. M. Wellman and E. S. Spelke. 2002. Infants‘ ability to connect gaze and emotional expression to intentional action. Cognition. 85(1): 53–78. doi: 10.1016/s0010-0277(02)00073-2)| false
Qu, C., W.-P. Brinkman, Y. Ling, P. Wiggers and I. Heynderickx. 2014. Conversations with a virtual human: synthetic emotions and human responses. Comput. Human Behav. 34: 58–68. doi: 10.1016/j.chb.2014.01.033)| false
Russell, J. A. and L. F. Barrett. 1999. Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. J. Pers. Soc. Psychol. 76(5): 805–819. doi: 10.1037/0022-3518.104.22.1685)| false
Sarnarawickrame, K. and S. Mindya. 2013. Facial expression recognition using active shape models and support vector machines. Paper presented at the Advances in ICT for Emerging Regions (ICTer).
Scherer, K. R., T. Bänziger and E. B. Roesch (eds). (2010). A blueprint for affective computing. Oxford University Press, New York.
Scherer, K. R. 1984. On the nature and function of emotion: a component process approach. In: (K. R. Scherer and P. Ekman, eds) Approaches to emotion. Erlbaum, Hillsdale, NJ, pp. 293–317.
Scherer, K. R. 2000. Emotions as episodes of subsystem synchronization driven by nonlinear appraisal processes. In: (M. D. Lewis and I. Granic, eds) Emotion, development, and self-organization: dynamic systems approaches to emotional development. Cambridge University Press, New York, pp. 70–99.
Scherer, K. R. 2001. Appraisal considered as a process of multi-level sequential checking. In: (K. R. Scherer, A. Schorr and T. Johnstone, eds) Appraisal processes in emotion: theory, methods, research. Oxford University Press, New York, NJ, pp. 92–120.
Scherer, K. R. 2013. The nature and dynamics of relevance and valence appraisals: theoretical advances and recent evidence. Emot. Rev. 5(2): 150–162. doi:
Scherer, K. R.and H. Ellgring. 2007a. Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion, 7(1): 113–130. doi: 10.1037/1528-3522.214.171.124)| false
Schwabe, L., C. J. Merz, B. Walter, D. Vaitl, O. T. Wolf and R. Stark. 2011. Emotional modulation of the attentional blink: the neural structures involved in capturing and holding attention. Neuropsychologia. 49(3): 416–425.
Schönbrodt, F. D. and J. B. Asendorpf. 2011. The challenge of constructing psychologically believable agents. J. Media Psychol. 23(2): 100–107. doi:
Vuilleumier, P., M. P. Richardson, J. L. Armony, J. Driver and R. J. Dolan. 2004. Distant influences of amygdala lesion on visual cortical activation during emotional face processing. Nat. Neurosci. 7(11): 1271–1278.
Vytal, K. and S. Hamann. 2010. Neuroimaging support for discrete neural correlates of basic emotions: a voxel-based meta-analysis. J. Cogn. Neurosci. 22(12): 2864–2885. doi:
Vytal, K. and S. Hamann. 2010. Neuroimaging support for discrete neural correlates of basic emotions: a voxel-based meta-analysis. J. Cogn. Neurosci. 22(12): 2864–2885. doi: 10.1162/jocn.2009.21366)| false
Wimmer, H. 1983. Beliefs about beliefs: representation and constraining function of wrong beliefs in young children‘s understanding of deception. Cognition. 13(1): 103–128. doi: 10.1016/0010-0277(83)90004-5)| false
Ye, W. and X. Fan. 2014. Bimodal emotion recognition from speech and text. IJACSA. 5(2).
Zeng, Z., M. Pantic, G. I. Roisman and T. S. Huang. 2009. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern. Anal. Mach. Intell. 31(1): 39–58. doi:
Zeng, Z., M. Pantic, G. I. Roisman and T. S. Huang. 2009. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern. Anal. Mach. Intell. 31(1): 39–58. doi: 10.1109/TPAMI.2008.52)| false
i-com - Journal of Interactive Media is devoted to human-computer interaction, media design, usability, engineering and systems evaluation, software ergonomics, cooperative systems, e-learning, mobile and ubiquitous systems, user-adaptive systems, agent development tools and methods for media in different application fields, barrier-free systems design, and the social aspects of information and communication technologies.