Human Capacities for Emotion Recognition and their Implications for Computer Vision

1 Introduction

In recent years, research has been increasingly concerned with the role of emotions during the interaction between user and interactive systems under the phrase affective computing (Picard 1997). The fundamental idea is that by “incorporating” emotions into interactive systems the interaction process would lead to more appropriate system responses (Gratch and Marsella 2004) and thereby to responses of the system user that are comparable to real situations (de Melo, Carnevale, and Gratch 2012, Krämer et al. 2013). Following these functions, affective computing systems have to fulfill two requirements: The system (1) needs to perceive emotional reactions by the user to successfully incorporate this information into the interaction process and (2) it requires the ability to generate emotional processes itself. While these two requirements are often considered as separate functions of artificial cognitive systems, we argue that both are strongly interconnected and necessitate each other. Thus, any system fulfilling one of those functions would automatically have the ability to fulfill the other function as well. To go even further, emotional information processing cannot be clearly separated from other domains of information processing, because they are highly integrated (Pessoa 2008, 2012, Okon-Singer et al. 2015). This argument rests upon the premise that affective capabilities of interactive systems can only be modeled appropriately, if they are modeled after the human cognitive system (Vitay and Hamker 2011, Canamero 2005). The idea behind this statement is the question, why we should expect artificial systems to perform equally well in the domain of emotion recognition compared to humans, although they follow a completely different logic of information processing.

In this article we will review the human capacity to perceive and process emotional information and present implications for the design of affective interactive systems that are capable of perceiving emotions in their users. We discuss the social functions emotions can serve in HCI and thereby demonstrate the importance of the endeavor to model emotional processes apart from being important for emotion recognition. Since we argue that affective computing as a basis for automated emotion recognition needs to resemble affective processes in humans, we review findings from psychology and neuroscience concerning the human capacity for emotional information processing. Therein, we focus on appraisal theories of emotion and the role of the amygdala in the early evaluation of intrinsic stimulus properties. To provide implications of how systems for automated affect recognition can benefit from this knowledge, we suggest a biologically inspired model of the amygdala. It is able to recognize appraisals of intrinsic stimulus properties in others as a first step towards biologically motivated models of emotion recognition. Finally, we give an overview about common methods for automated affect recognition and discuss shortcomings of these approaches concerning our suggested model and the properties of the human neural circuit.

2 Affective Computing and HCI

Affective computing has been widely discussed in the domain of embodied conversational agents (ECAs; e. g. Gratch and Marsella 2005, Becker-Asano and Wachsmuth 2009, Scherer, Bänziger and Roesch 2010), which are capable of natural interaction. But why should researchers want to consider emotional processes for their virtual agent architecture? Since ECAs aim to emulate human behavior, we have to consider the functions emotions serve in humans. The intrapersonal functions of emotions in terms of their contribution to evolutionary adaptiveness have been widely discussed (Frijda 1986, Lazarus 1991, Levenson 1999). Yet, these functions are only of secondary importance for the most part of HCI research. In line with the emotions as social information model (van Kleef 2010) emotional processing and expressions serve a number of interpersonal functions in human communication both on the side of the receiver and the communicator. First, they allow us to better understand our social environment by observing emotional reactions of others towards and understanding their appraisals of third entities (Manstead and Fischer 2001). Thereby, emotional expressions can help disambiguating social situations (van Kleef et al. 2011) and may also inform us about the affective value of novel entities, resulting in approach or avoidance behavior. Second, emotional expressions of others allow us to understand their intentions and predict their behavior (Phillips, Wellman and Spelke 2002). This theory of mind (Premack and Woodruff 1978, Wimmer 1983) is a fundamental ability that evolved out of environmental pressures towards bigger social groups and constantly updates presumed mental states of others. Third, emotional expressions communicate information about relationships or social status (Fischer and Manstead 2008) and serve group and individual differentiation (Levenson 1999). As a consequence, this wide array of social functions renders emotional expressions important to better understand others and our mutual environment. ECAs might use the affective information from the user to understand their evaluation of situations and could thereby learn the affective value of novel stimuli, and vice versa. The absence of emotional expressions that would otherwise fulfill these functions can have detrimental effects on the interaction process. Thus, when technology is not capable of conveying affective information correctly, the user might be prone to misinterpret an ECA’s behavior due to misperceptions of affective states (Liebold and Ohler 2013). This might at the same time result in unfavorable evaluations of the ECA. Qu and colleagues (Qu et al. 2014), for example, found that random expressive behavior of virtual agents during conversation decreased conversation dissatisfaction.

Following the social functions of emotion expressions, affective computing in interactive systems has been shown to have positive effects on the interaction process. In his model of social influence, Blascovich (Blascovich 2002, Guadagno et al. 2007) proposed that agency and behavioral realism are fundamental constituents of social influence of ECAs. Emotional processes are relevant for both dimensions, because emotionally driven behavior might increase behavioral realism and might thereby also facilitate agency believes, which are deduced from observed behavior. The increased social influence of ECAs with affective computing capabilities has been demonstrated in several studies: De Melo, Carnevale, and Gratch (2012) found that emotion displays of an ECA influence user decisions during virtual negotiations. Regarding the underlying mechanism, they showed that users perceived the ECA’s emotion displays as indicators of its behavioral intentions. Regarding mimicking behavior, Krämer, Kopp, Becker-Asano, and Sommer (2013) reported that users smiled longer over the course of a communication episode, when the ECA smiled at times. At the same time explicit evaluations were not different from the non-smiling ECA. Other research on the link between affective computing and user interaction focused more on the perception of ECAs. Demeure, Niewandomski, and Pelachaud (2011) found that emotional reactions of ECAs in general led to higher evaluations of their competence, warmth, and believability. The effect was greatest for emotion displays that were both appropriate and plausible — yet, even inappropriate and implausible displays led to better evaluations. In another study addressing agent evaluation, Bosse, and Zwanenburg (2009) reported better evaluations of ECAs that were capable of producing emotional reactions.

Apart from these results, it was also found that nonverbal behavior outweighs the positive effects of emotion displays during interaction processes (Cassell and Thorisson 1999, Krämer, Iurgel and Bente 2005). Yet, one should not conclude that researchers and developers should instead focus on aspects of nonverbal behavior completely. The hard distinction between emotion and other nonverbal behavior does not consider the fact that emotional processing can inform the selection of appropriate nonverbal behavior, such as nodding (Lee and Marsella 2010). Thus, by modeling emotional processes as a fundamental mental capacity ECAs may cover a wider range of interaction scenarios than pre-defined conversational functions (Schönbrodt and Asendorpf 2011). Some ECA scenarios even require affective computing, such as virtual patients, which aim to train novice therapists conversational and diagnostic skills (Kenny et al. 2007, 2008). Additionally, apart from specific interaction contexts, modeling emotional processes is important from a research perspective, because it allows emotion researchers to test their theoretical predictions and thus advances our understanding of emotional processes in human cognition.

A core component of affective ECAs is their ability to affectively interact with the user rather than merely to produce emotional reactions. To achieve this goal, they need to recognize the user’s affective states in the course of a conversation. While several systems for automated emotion recognition have been presented (for a review see Zeng et al. 2009), these approaches usually consider emotion recognition as an independent system, which categorizes emotional displays either using handcrafted rule-based detection or some kind of machine-learning algorithm. Research in emotion psychology and on the neural basis of emotional processing, however, suggest that emotion recognition is a highly integrated process that relies on more fundamental capabilities of emotional processing. Therefore, any emotion recognition system aiming for human-like capabilities needs to reflect affective processes in humans requiring interdisciplinary cooperation between computer science, emotion psychology, and cognitive neuroscience.

3 Human Emotion Processing

A great number of theories have been presented in the field of emotion psychology to answer James (1884) question: What is an emotion? The presented theories differ in their focus and the proposed mechanism, how emotions are elicited, and consequently in their definition of the construct of emotion. Although up to today no consensus about these questions has been reached, most researchers agree that emotions are syndromes of subjective feelings, psychophysiological responses, actions tendencies, and some form of cognitive processing (Frijda 1986, Oatley and Jenkins 1996, Izard 2010, Mulligan and Scherer 2012). Yet, the nature of the involved cognitive processes remains subject to debates. Some researchers argue that emotions are evolved affect patterns or basic emotions that met specific evolutionary pressures (Ekman 1972, Izard 1977, Plutchik 1980) and therefore exist as distinct neural activation patterns (e. g. Vytal and Hamann 2010). Others argue that emotions are a result of more fundamental cognitive processes (automated situation appraisals), which are by themselves evolutionary adaptive (Scherer 1984, Lazarus 1991). A third position is that different emotion words are merely a result of language development and categorization of affect and that emotions in principle only differ along either two or three fundamental dimensions, which are valence, arousal, and dominance with the first two being generally agreed upon in this perspective (Mehrabian and Russell 1974, Russell and Barrett 1999). A recent extensive meta-analysis on the neurophysiological evidence for these approaches was presented by Lindquist and colleagues (Lindquist et al. 2012). They found that the findings lend more support to appraisal theories than theories that assume distinct neural activation patterns for each emotion.

3.3 A Brain-Inspired Model of Affect Recognition

Implementing these parallel and automatic processes into a machine recognition system will enable us to significantly increase the appropriateness of system responses in human-machine interaction. Further, it can at the same time be a helpful approach to solve the bottleneck problem of processing all the information of the system’s environment by focusing on the relevant information. Recently, fundamental progress has been made within the field of computational neuroscience to facilitate the implementation of such a circuit. Pessoa and Adolphs (2010) describe the neuroscientific foundations for a potential architecture of such a system. Other groups have examined the computational microcircuits for processing visual information and included attentional feedback (e. g. Beuth and Hamker 2015). This research on attention circuits in the visual system demonstrates how the visual cortex integrates feedback signals to obtain attentional modulation for stimuli. Further, this can be used to integrate other kinds of sources for attentive feedback into the visual processing. Accordingly, in such a circuit the amygdala would play the role of a higher area receiving information from the cortex and modulating it reversely through its feedback.

To simulate the amygdala, it is important to differentiate between at least two emotionally relevant parts of the amygdala. First, the basolateral amygdala (BLA) receives input from various human senses (McDonald 1998, Sah et al. 2003). It can connect a salient stimulus with an already emotion-laden stimulus (Bergstrom et al. 2012) and feed back to the visual cortex (Amaral, Behniea and Kelly 2003, Vuilleumier et al. 2004). Second, the central amygdala (CE) receives information from BLA (Pape and Pare 2010) and modulates the current body state through its output connections (LeDoux et al. 1998). By this basic fragmentation, the amygdala converts the incoming information into a body reaction without adapting to a specific emotional state.

Based on the current understanding of the involvement of the amygdala into affective processes and its structural integration into the brain we propose a conceptualization for a model of visual emotion processing. Considering the previously given arguments, the model would be able to learn intrinsic stimulus properties, use these processes to understand facial expressions of emotion, and direct attention at emotionally relevant stimuli. The model should consist of the important ventral visual areas V1, V2, V4, and the inferior temporal cortex (IT), with their typical hierarchical order and bidirectional connections with each other. With this connectivity pattern the feedback from higher areas can be propagated back to lower ones. The amygdala, on the other hand, consisting of BLA and CE should be bidirectionally connected to V4 and IT to modulate the visual processing. Furthermore, BLA should receive input from PFC to regulate BLA activity according to the valence of the stimulus, which is encoded by PFC based on its input from IT (see Figure 1).

Figure 1

Conceptual structure for a hierarchical model of human-like emotion processing. BLA: basolateral amygdala, CE: central amygdala, IT: inferior temporal cortex, PFC: prefrontal cortex, V1, V2, V4: visual areas 1, 2, 4.

The model is capable of learning intrinsic stimulus properties and thus emotional relevance detection by conditioning the stimulus with body reactions in BLA. After several repetitions of the stimulus conditioning, the model will automatically increase arousal by triggering body reactions representing the appraisal of intrinsic pleasantness and goal-conductiveness. This mechanism also accounts for the detection of emotionally relevant facial features. As soon as the model has learned relevant facial features, the aforementioned appraisals would be automatically activated in tandem, which can be considered as early affect recognition. At the same time, BLA-activity is fed back to higher areas of the visual cortex to facilitate attention distribution towards the conditioned stimulus. This behavior is consistent with the novelty appraisal proposed by Scherer (2013). Further, the hedonistic value of the stimulus can be used by PFC to modulate amygdala activity. Thereby, positive evaluations of an event will inhibit amygdala activity.

4 Comparing Human Emotion Recognition and Current Computational Models

In order to give interactive systems the ability to interact with the user on an affective level, a broad variety of affect-recognition models have been developed over the last decade. The analysis of emotional facial expressions is the most common form of automated emotion recognition (Calvo and D’Mello 2010), since it is a very reliable emotional cue with a rich body of related research. Nonetheless, by interpreting the prosody features of spoken words or the body posture of a person, similar information can be obtained and several models have been developed for these kinds of expressions (Zeng et al. 2009, Kleinsmith and Bianchi-Berthouze 2013, Ye and Fan 2014, Javier et al. 2015). While visual and auditory cues are usually the most important sources of information for human-human interaction, in case of HCI the sources can vary more widely. Data can also be acquired by measuring electrical brain activity with an electroencephalograph (EEG), brain imaging through functional magnetic resonance imaging (fMRI), or recording of vital signs, and be used in computational models to perceive the emotional state of the human counter-part (Calvo and D’Mello 2010). Yet, these approaches are based on information that are typically not available in HCI contexts and would greatly impair the usability of interactive systems. Thus, we focus on visual models and particularly models for facial expression recognition.

Models extracting emotions from human faces are numerous (Bartlett et al. 2006, Gunes and Piccardi 2007, Li et al. 2013). An extensive survey as well as an overview of related reviews has been presented by Zeng, Pantic, Roisman, and Huang (2009). Multiple computational models were found to perform reasonably well on images and video streams (range between 39.58 and 95 depending on the test setup). In most cases these models categorize emotions either by machine learning algorithms or rule-based approaches. Common methods are the tracking of facial points and relating them to facial action units (e. g. Kotsia et al. 2008, Valstar and Pantic 2012) or the extraction of shape features (e. g. Sarnarawickrame and Mindya 2013, Lozano-Monasor et al. 2014) as Gabor wavelets, followed by a classification, for example through support vector machines (e. g. Whitehill et al. 2009, Piatkowska and Martyna 2012, Lozano-Monasor et al. 2014). To deal not only with static expressions, some models use the output of the classifier as input to an hidden Markov model to analyze the temporal appearance of an expression (e. g. Barlett et al. 2003, Koelstra, Pantic and Patras 2010, Lahbiri et al. 2013). Even neural networks have been used for emotion recognition as alternative approach for extracting sophisticated shape features (e. g. Prevost, Belaroussi and Milgram 2006, Senechal, Prevost and Hanif 2010), as direct emotion classifiers (e. g. Perikos, Ziakopoulos and Hatzilygeroudis 2014), or even for the whole facial analysis (e. g. Kukla and Nowak 2015). Neural networks can also be used to integrate features from different information channels. Yet, the proposed models differ greatly in their classification of affective states and approach to facial feature selection (between 2 and 9 emotion categories according to Zeng et al. 2009).

All these approaches differ substantially from an implementation based on neuroscientific evidence. Certainly, the visual cortex can be seen as a feature extracting stage. However, the amygdala is not merely a classifier to categorize the visual input. As previously explicated, the amygdala links visual information to body signals, whereby no classification in the sense of categorizing stimuli into discrete classes (e. g. emotions) has to be carried out. Consequently, there is no explicit decision for a particular kind of perceived emotion —  rather, a stimulus invokes a previously learned response according to its intrinsic emotional valence. Since we follow a more fundamental approach where stimulus and response are associated through learning, visual features do not need to be mapped on distinct classes. Thus, whenever a learned stimulus is detected, a reaction is immediately triggered.

Whereas, current computational approaches remain limited to a few easily distinguishable emotional expressions that are usually considered to represent basic emotions, a neuroscience-inspired model of emotion recognition can account for more fundamental emotional processes. In our model, these processes are responsible for detecting emotional relevance (novelty and intrinsic pleasantness) of a stimulus as an early step in emotional processing without higher cognitive involvement. While it could be argued that this is essentially a step back from identifying more or less concrete emotional feelings, this approach has several advantages. It is inherently plausible, because the goal of all approaches is to mimic the human capacity to recognize emotions and the most plausible means is to mimic the human brain. Once such a model has been developed, it can be easily implemented together with other models of human cognition. The good compatibility of these models results from the fact that the account for universal functions of human cognition. They can complement each other, but do not fundamentally change with the context. As a consequence, they can also be applied in various contexts and are not limited to special cases, such as emotion recognition from facial features. In fact, the proposed amygdala model covers both the production and recognition of emotional processes and is independent of the modality, in which emotional displays of the interaction partner are encoded. Lastly, modeling emotion recognition as appraisal recognition accounts for the complex environmental conditions that lead to an emotional expression in others. An angry face is never merely an angry face, but an indicator of an arousing stimulus of negative and sometimes norm-violating quality, for which the other person is confident that she is able to overcome the problem. Understanding intrinsic stimulus properties is the first step towards the necessary appraisals for these kinds of judgments.

Since the proposed processes need to be modeled as brain-inspired neural networks, this type of model comes in tandem with other beneficial features. First, the mechanisms to learn associations between stimuli and reactions are directly implemented into the model. Thus, the model will not have to reach a final state, which is then implemented into an interactive system —  rather the model is still capable of learning associations after it has been implemented. Second, emotional processing actively influences feature extraction in the visual cortex. It directs attention to emotionally relevant features and entities, facilitating their processing (Adolphs and Spezio 2006). This mechanism should prove useful for HCI as well, when only limited resources for scene analysis are available and a more precise analysis of relevant features is necessary.

5 Conclusion and Challenges

Integrating emotional processes into HCI technology is an important research field to render interactive systems capable of more appropriate system responses to emotional user behavior. An important starting point for these endeavors is the recognition of affective user states. Although, numerous methods for the automated recognition of facial expressions have been presented, the methods cannot match the human ability to recognize emotions and incorporate them into interactions. As a first step to overcome these limitations, we suggest a substantially different approach to common emotion recognition systems. Instead of categorizing emotion indicators (i. e. facial features) into distinct emotion categories we suggest modeling emotion recognition after the human capacity for emotional processing in general. This is necessary, because the models for emotion recognition aim to mimic outcomes of human information processing and the involved mechanisms in humans are responsible for both emotion elicitation and recognition. This goal can be achieved by incorporating neuroscientific evidence on the visual cortex-amygdala circuit and related findings of emotion psychology. In the proposed model visual information is conditioned to automatically elicit body reactions and feedback to higher visual areas to enable affect-based attention allocation.

This approach has the advantage of not being limited to few distinct and complete facial expressions but tackling the problem at the level of single facial features. Furthermore, the association to body signals can be learnt during real interaction settings keeping the system flexible, when employed in new contexts. The model in general is not restricted to be used only for emotion recognition, because it also generates visual output modulated by emotional relevance at the same time, which can be used for further processing and decision stages. These stages do not necessarily need to be modeled as neural systems considering the current lack of advanced neural models for higher cortical areas and processes, respectively. Yet, the resulting visual information and appraisals of our model can be integrated into existing computational architectures for recognition, decision-making, or higher-level emotional processing (e. g. Becker-Asano 2013) without strict adherence to the neurophysiological basis. This way, more complex affective processes can be modeled without the need of having a complete understanding of the interactions between involved brain areas —  which we simply do not have at the moment. In such integrated systems, the emotional feedback in our system could, for example, still guide attention allocation to the relevant scene information, easing the bottleneck problem of scene analysis. Consequently, our proposal focuses on modeling early stages of emotional processes and should provide a sound basis for further efforts to model artificial emotions in line with automated emotion recognition and expression.

This approach leaves several aspects of human emotions and their technological implementation untouched. (1) Human emotions are oftentimes a result of more complex situational appraisals. However, these processes rely on more complex interactions within our neural system involving at least memory and conscious reasoning. Thus, we have focused on appraisals of intrinsic stimulus properties, accounting for emotional processes, upon which subsequent appraisals are based. These can only be modeled given that we have robust models of the necessary fundamentals. (2) Our approach focuses on appraisal theories. While compelling evidence for this viewpoint has been presented, not all emotion researchers agree that universal situation appraisals are the foundation of emotional processes. The basic emotion approach stating that emotions are based on evolved, but distinct, neural circuitry specific to each emotion has received reasonable attention both in psychology (e. g. Ekman 1972, Izard 1977, Picard 1997) and in the neurosciences (e. g. Phan et al. 2002, Murphy, Nimmo-Smith and Lawrence 2003, but see Lindquist et al. 2012). (3) Emotions are expressed as multimodal arrangements of several nonverbal channels including facial expressions, gestures, body posture, behavioral patterns, and tonal changes in the voice. Moreover, merely relying on facial expressions might lead to misclassifications, since social norms and deceptive intentions may lead to the suppression of emotional expressions. (4) It remains uncertain, whether emotions occurring in natural settings always achieve the desired intensity threshold in their expression that is required to achieve robust tracking results. Automated emotion recognition algorithms usually perform better under lab conditions compared to real world encounters, where emotion expressions occur less prototypically. More so, expressions might as well be very subtle rendering it hard to classify them as significant indicators of emotional processes. Possibly, multimodal feature extraction might increase tracking reliability under these circumstances. (5) Since brain-inspired models of emotion recognition involve the simulation of complex models of neural networks, it remains a challenge to scale these models appropriately both to be capable of real-time processing and perhaps even to cover low-budget solutions with even less processing power.

Future work should further focus on implementing computer models for early stages of emotional processing. Although we believe in the benefits of this approach, first prototypes have to assess their ability for affective processing using standardized as well as natural datasets. As evidenced in this article, these problems need to be tackled by interdisciplinary research groups from psychology, cognitive neuroscience, and computer science to incorporate findings from various fields to create more lifelike and engaging interactive systems for HCI.