The Interactive Self – A Review on Simulating Social Interactions to Understand the Mechanisms of Social Agency

1.1 The Minimal Self and Agency

The so-called ‘minimal self’ is a common approach trying to describe the most basic experience of selfhood. This fundamental feeling of self-awareness is believed to be connected to two main concepts: Body ownership and the sense of agency. Body ownership describes the sensation that one’s own body belongs to oneself [1]. Agency on the other hand is linked to the implicit coupling of action predictions and a perceived outcome (e. g. [2]). It has been intensively studied within action and motor control research as a phenomenon caused by interactions with our environment. Theories of agency assume that it is based on internal predictive sensorimotor cues comprising the action plan and the comparison to external sensory cues leading to a postdictive implicit sense and explicit judgement of being responsible for an action [3]. In addition, cognitive aspects such as the type of environment, the outcome valence and the affective state of the individual [3], [4], [5] are likely to influence the experience of agency. Such experiences of being in control of self-generated actions and the consequences can be measured in an implicit or explicit way. One commonly used implicit concept is intentional binding, which assumes that self-produced actions are perceived as being in closer proximity of the related outcome compared to externally generated stimuli [6]. In an explicit procedure, agency is usually measured by a conscious judgement of experienced control over an action consequence (e. g. [7]).

1.2 The Concept of Social Agency

Transferring the concept of agency into the social domain, sensory cues, social context as well as social norms are highly important for judging one’s own action-ownership within social interactions [8], [9], [10], [11]. In a more principled manner it has been asked (see [12]) whether the minimal self should be considered either as a built-in feature of our experiential life (e. g. [13], [14]), or whether it is a later acquisition emerging via social exchanges and interactions (e. g. [15], [16], [17], [18], [19]). Emphasising the social interactive dimension but surpassing a dichotomy between the self and the other [12], the self can be regarded as a continuous and collective process of dialectical attunement between the organism and the social world across multiple scales [20], [21]. In this article, we will focus on ‘social agency’, which can be defined as the sense of self that is gained through the perceived control one exerts over the social world. In this account, agency is not limited to one’s own motor actions, but is also implicated in social interactions and in receiving feedback from other individuals. To put it simply, selfhood is developed through the (conscious or unconscious) recognition that one actively and reciprocally affects the social world (cf. [21]). Wolpert et al. [22] have proposed a computational framework that extends the action control model to the domain of social interactions. This framework claims that individual motor control and social interaction contain analogous processes including a control signal (an action plan producing either a pure motor act or on the other hand a communicative action), an action outcome (which can be a bodily reaction or a reaction in another person) and the perception of these outcomes ([22]; Figure 1 A, B).

Figure 1

A–B Illustration of analogy between the phenomenon of agency within action control and social interactions. When an action is performed (either pressing a button as in A) or looking at someone (B) outcome predictions are made and followed by action-outcome associations when the outcome is perceived. C) Mind attribution and belief of agency are an elemental process of social agency and also occur within virtual reality setups (VR: virtual reality).

The common central aspect, which is one of the core elements for the phenomenon of agency, is the experienced relatedness of one’s own action-outcome predictions to the perceived outcome in the environment. Despite the clear analogy in this process, research so far has mainly focused on individual motor actions in order to study the sense of agency. This is surprising, given that general self-representations, self-other distinctions and outcome-predictions are primarily and ontogenetically formed during social encounters [17]. Already very early in life, humans learn that looking at something or someone can cause a reaction in another person [23], [24] and such action-outcome associations comprise a core element of the phenomenon of agency. Only a small number of studies so far have looked at the experience of agency within social interactions, i. e. the sense of social agency [25], [26], [27]. Recently, Stephenson et al. ([27]) applied a well-established paradigm based on the concept of intentional binding to also include social cues. This paradigm was based on the idea that we perceive a shorter timeframe between actions (e. g. one’s own gaze) and outcomes (e. g. the gaze of another person) of self-generated actions compared to two events that were not caused by oneself. The study showed that after initiating a gaze shift, gaze following (indicated by a picture of a face) could elicit an implicit sense of agency as measured by the intentional binding effect as well as an explicit experience of control. The relatedness of implicit and explicit measures of agency has been a topic of discussion in previous action control studies (e. g. [7], [28]). In the study by Stephenson et al. [27] it was shown that an implicit measure and the explicit judgment of control are closely related for social agency. Overall, this is an essential study, because it demonstrates the existence of agency not only in the realm of motor control, but also within the social domain. Yet, this study is still limited in terms of the ecological validity of the used method. While it stresses the relevance of social gaze cues for the experience of social agency, it does not represent a truly interactive social situation. This is problematic, as it has recently been emphasised that it is important to study the relevant phenomena in a realistic and ecologically valid manner in order to grasp their full complexity [17].

One’s own experience of social agency depends on different factors. We propose that the reactivity shown by another person within a real-time social interaction strongly influences one’s own experience of social agency. More precisely, we expect that the responsivity is likely to be dependent on the temporal contingency of action-outcome coupling [26] as well as the congruency of the social cues [25], [27]. Within social interactions, the latency of responses has to lie within a certain time frame in order to induce an experience of relatedness between one’s own action and another agent’s response to it. Congruency on the other hand refers to the outcome type and whether it is perceived to be ‘in line’ with one’s own action or predictions: Gaze following within a social interaction as well as direct gaze for example, compared to averted gaze could be considered a congruent response. This parameter is also strongly related to the valence of the outcome, since research has shown that congruent social behaviour such as direct gaze and gaze following is experienced to be more pleasant [29], [30], [31]. The sense of agency has been shown to be related to outcome valence [4], [5] and a similar influence can be expected for social agency [31]. The experimental modulation of the contingency and congruency of social cues occurring in response of a person’s action can therefore be a useful tool to manipulate and study social agency. Particularly, due to human’s high sensitivity to eye movements of others [32], gaze cues are a powerful parameter to investigate in this context.

Another important factor in the analysis of social agency is the belief that the interaction partner is intentional. Research suggests that humans can sense agency in others and that the belief of the others’ agency and mind also influences the processing of the perceived behaviour [33], [34], [35]. Indeed, it has been suggested that perceiving others as intentional agents might be the first crucial step ontogenetically towards unique human social cognition [36]. Behaviour presented by a computer-controlled agent for instance is expected to be different from the one shown by real humans within an interaction [25], [31], [35], [37]. Therefore, if we assume that one’s own experiences of social agency are based on the expectations we have for social reactions by others, it is necessary to consider that these expectations are likely to be dependent on the beliefs we have about the agent with whom we are interacting (Figure 1 C).

While considering the fundamental importance of social agency to the formation of the self and self-experiences, in this article we aim to review and suggest possibilities of how to enrich this field of research with virtual reality technology, in particular a solution that allows to simulate a gaze-based social interaction. Furthermore, we describe a predictive coding model of social agency, which will allow for a formal study of competing hypotheses about relevant mechanisms. In the end, we emphasise the prospects of an approach that brings together virtual reality and computational modelling for understanding social agency across different psychiatric disorders.

2.1 Gaze-Contingent Paradigms

In the field of behavioural research, an interactive method was developed using responsive anthropomorphic virtual characters to simulate social interactions [30], [53], [54], [55]. Such gaze-contingent paradigms use eye-tracking in order to measure the participant’s gaze behaviour and control the reaction of the virtual character relative to the participant’s gaze. This allows the researcher to simulate a gaze-based social interaction by controlling the gaze behaviour of the simulated interaction partner with a computer algorithm according to various experimental conditions. Usually, these experimental procedures include a so-called ‘cover story’ telling the participants that the gaze behaviour they see from the virtual character is controlled by a real person also sitting in front of an equivalent eye-tracking system (e. g. [30]). This method allows analysing behavioural and neural mechanism of processing gaze behaviour such as gaze following, mutual gaze or gaze dynamics in a real-time social interaction [17], [30]. More particularly these methods allow to control and experimentally manipulate the responsiveness of a simulated interaction partner in the virtual reality platform and can, therefore, systematically manipulate the sense of social agency elicited within the social interaction.

As mentioned before, a crucial parameter modulating the experience of social agency is the temporal contingency between the action and the associated social reaction. According to Pfeiffer et al. [26] the latency of a gaze reaction needs to fall into a certain temporal range (around 400 ms) to be perceived as contingent upon one’s own actions, leading to the experience of relatedness and being addressed by the person directly [25], [26]. Next to the temporal dynamics of gaze, also the gaze direction influences the experience of relatedness of another agent. Congruent gaze behaviour such as gaze following compared to averted gaze as well as certain proportion of direct gaze seems to be expected in interactions with humans [25], [26], [56], [57]. Furthermore, these expectations or predictions one makes about the reactions of another agent are strongly dependent on the social context and the goal of the interacting agent (e. g. cooperative, competitive or undirected) [25], [26], [35]. Since the general experience of agency is dependent on the outcome predictions, it is necessary to take into account possible differences in the attributed intentions of humans compared to non-human virtual agents when looking at social agency. While gaze-contingent experiments demonstrate a prospect of simulating a real-world situation and modulate the responses in these situations in order to study the experience of social agency very well, it is necessary to examine and evaluate if these methods are suitable to simulate a believable interaction with a human.

2.2 Interacting with Virtual Agents

Overall, research shows that the interaction with virtual agents was found to be perceived similarly to an interaction with human-controlled avatars [58] and also comparable regarding relevant determinants such as social presence and intimacy [59]. Also the manifestation of social inhibition and facilitation, interpersonal distance regulation as well as empathy and prosocial behaviour did not differ from the one observed with human interaction partners in some studies [51], [60], [61], [62], [63]. The reason for that might be human hard-wiring to respond to cues that indicate the presence of an intentional agent [64], [65], [66]. Nonetheless, some aspects seem to be critical for the simulation to succeed: An agent will be most likely ascribed a mind if it appears to have goals, which it pursues with its own actions. Further, the agent has to be reactive to a person’s behaviour and respond adequately [67]. Mind-ascription is central in creating the impression of a real personal interaction [68], [69], [70], [71], has an impact on the subject’s behaviour [72] and most likely also on the experience of relatedness of one’s own action to the reaction of an interaction partner. When an entity is perceived to have a mind, actions are interpreted to reflect internal states such as emotions or intentions [68], [70]. Furthermore, entities that do not trigger mind perception have negative effects on performance in social interactions [73], [74], [75], and fail to induce social facilitation effects [76], [77]. It can be speculated that differences in mental state attribution also affect the outcome predictions within a social interaction, leading to possible differences in sensed social agency. When an interaction partner is assumed to be human, intentions are automatically ascribed (see [73], [74], [75]), but it can be argued, that the ascription of humanness is influenced by the appearance of the agent. Especially, concerning appearance and motion of computer-animated agents or robots a disparity is still visible and has a negative effect on the social interaction. The more human an agent looks, the stronger the social response to it (e. g. [78], [79]). Along these lines, a lack of human appearance diminishes mental state ascription [70], [71]. Appearance judgements seem to primarily take an agent’s facial features into account with eyes being disproportionately informative compared to other physical features [80]. In summary, there should be an aim for a more human look, to ensure that a mind is being attributed to an agent. Further development of agents’ appearance as well as motion patterns presents itself as the next step in increasing the ecological validity of simulated social interactions. Notably, the progress should take place at the same time, since a mismatch between a human-like look with non-human motion elicits distress, referred to as the uncanny valley effect (e. g. [81]).

2.3 The Experience of Presence in Simulated Social Interactions

A related and highly relevant concept in this respect is the experience of presence within virtual environments, which is a commonly studied phenomenon in telecommunication and virtual reality research [39], [82]. Most commonly, presence within a simulated reality is defined as the sensation or experience of being present in the virtual environment [39], [83], [84], [85], [86]. This definition already includes the aspect of transportation, which refers to the extent to which people feel transported to and located within the new environment [82], [87], [88], and the perception to be embedded in an environment that provides a continuous stream of stimuli and experience [83]. Importantly, there is also a social component to presence in particular when a social interaction is mediated within a virtual reality. Social presence is defined as the sense of being together [87], [89], [90] and having access to another mind [65]. The concept appears to be quite interleaved with mind perception or attribution as described before. Whereas the definition of mind perception is closely related to intention ascription and thus comprises more of a cognitive component, social presence implies a more affective aspect. Rüggenberg et al. [90] describe the latter as emotional closeness and social relatedness. Short et al. [91] mark it as “the degree of salience of the other person in the interaction and the consequent salience of the interpersonal relationships” (p. 65). Social presence is also characterised as a rather unidirectional concept. Here, it is assumed that it is only relevant that a person perceives another mind. This distinguishes the construct from co-presence, which is described as sense of being together and having access to each other’s minds in a bidirectional way [92], [93]. Subjects have to feel like they are able to perceive their interaction partner and further that their interaction partner can actively perceive them [65], [87]. Co-presence thus denotes the sense of ‘connection with another mind’ and includes both the perceived involvement of the partner in the interaction and the self-reported own involvement in the interaction [65], [87]. Therefore, co-presence is highly essential in studying social agency in virtual realities since the expectations of a social response will be strongly influenced by the knowledge of another person’s involvement or co-presence within the interaction. Participants should have a sense of being involved in the interaction themselves and perceiving the other as well as being perceived and responded to at the same time.

Taken together, it can be stated that computer-based simulations are extremely useful to study feedback processes that are constitutive of social interactions while providing a high level of experimental control. However, the limited design of some virtual agents used in research may also hamper the ecological validity of studies and can possibly influence core processes of perceived social agency caused by the mentioned differences in mind attribution. Naturalistic systems may help to resolve this issue to some extent [50], but the problem remains that so far even naturalistic virtual agents are still not perceived as real humans. One possible solution to this problem is to simulate a continuous social interaction by including natural stimuli of human agents. Instead of a virtual character, a real recording of a person’s behaviour can be presented in a continuous and gaze-contingent way to create the illusion of a photorealistic, real-time social interaction [94]. Crucially, the pre-recorded video stimuli have to be pre-processed in order to reduce large motion jumps during successive display. Such stimuli can be controlled in the same way as anthropomorphic virtual characters, but still include small naturalistic movements and cues such as blinking and facial twitching. By presenting the recorded gaze behaviour with altering temporal delays and displaying different gaze directions in relation to the participants’ behaviour, it is possible to systematically investigate the experience of social agency. We propose that the integration of social gaze cues together with internal predictions will result in the sense of being the initiator of a social reaction and will therefore induce a sense of social agency. A video-based simulation can provide a high level of realism while avoiding the uncanny valley and eliciting the perception of being unmediated and believable. It further sets up a social context inducing the feeling of being addressed by another mind, which is necessary to study self-experiences through social agency in an utmost naturalistic way.

3.1 Describing Social Agency with Predictive Coding

As mentioned before, agency is believed to depend on an implicit coupling of actions and outcome perceptions [95]. When looking at social interactions the percieved outcome comprises different social cues which can, based on contingency and congruency, influence the experience of social agency. It is still unknown how these different cues are integrated and might contribute to different intensities and qualities of agency. One long standing account describing the sense of agency during action control is the so-called comparator model [96], [97], [98], [99], [100]. This model, which was predominantly used to describe agency within motor actions, suggests that intended and executed motor commands create internal forward projections which predict the perceived sensory outcome of the action (also referred to as motor predictions). In the comparator model, congruency between both predictions and sensory input lead to the attribution of agency to oneself [2], [101], [102]. More recent theories aim to improve and extend this model, since it might not offer a complete characterisation of the underlying mechanism of agency [100], [103], [104], [105]. Indeed, the predictive coding account has been advanced to provide a mechanistic account extending the original comparator model to a more general principle of brain organisation [106]. It has been proposed that the predictive coding framework offers a plausible description of the putatively underlying neural mechanisms of basic self representations as well as interoception [107], [108] and therefore appears as a most promising method for studying the perceptual integration which leads to the emergence of social agency.

Put simply, according to this theory, the long-term minimisation of the so-called prediction errors is the driving factor of all brain processes. Prediction errors in the brain can be described as the incongruity between perceived sensory information and calculated predictions, based on prior experience [109], [110], [111]. Importantly, beliefs are adjusted simultaneously across different hierarchical levels within the brain. Essential are two parallel processes happening at the same time: this includes the ascending prediction error in the hierarchy, which adapts the system in order to optimise predictions, while predictions descend the hierarchy resolving the prediction errors. Perception, learning and action can all be considered as various ways an organism employs to optimize prediction error minimization. Perception and learning can be thought of adjustment of the organism itself across different time scales, while action reversely is considered as the active modification and sampling of the environment by the organism, in the effort of aligning personal and environmental states. The hierarchical structure of this system is highly relevant, because it implements the inclusion of multiple levels of increasing abstraction. The latter might be exactly what enables a person to embody social interactions across multiple levels and deploy generalised forms of it in future interactions but also privately [21].

Drawing on this account, we hypothesise that the sense of social agency is dependent on internal predictions about expected social reactions of others (Figure 2 box in dark blue) and the actual incoming sensory information (Figure 2 box in light blue). A match of social expectations and the perceived social cues results in small prediction errors, which enhance the sense of social agency. A mismatch on the other hand results in high prediction errors, which diminishes the sense of social agency. Figure 2 aims to illustrate this mechanism schematically: Both the implicit feeling and explicit judgment of social agency is resulting from the mentioned underlying processes regarding the predicted and perceived social responses. Beliefs are updated across different hierarchical processing levels of abstraction (light grey circles) by ascending prediction errors (PE), while predictions (P) descend the hierarchy reducing the prediction errors. Other factors such as the context, experiences and prior believes about the interaction partner can influence this process.

Figure 2

A simplified schematic model of the underlying mechanisms of social agency. Both the explicit and implicit aspects of agency are based on prior expectations on social responses and the actual perceived response. High prediction errors (PE) lead to low social agency, while a match between predictions (P) and sensory feedback result in the experience of social agency. The process can be influenced by different factors on different levels.

Figure 3

Hierarchical Bayesian model of social agency (SA; defined by μ_SA and σ²_SA). The first family of states (x) tracks the sensory input of interpersonal congruency (u_x; e. g. perceived gaze following by the other), while the second one (y) predicts the sensory input of interpersonal contingency (u_y; e. g. the latency of gaze following), depicted in the blue box. The precision (π_x and π_y) weighted inverse-prediction-errors (IPE) are integrated on a subject specific fashion (via the weighting factor w), yielding the expected value of social agency (μ_SA). The variance of social agency (σ²_SA) is modelled via a free noise parameter ζ, which is divided by the product of π_x*π_y, so that it decreases when the agent is more confident about their social expectations. Adapted from Bolis et al. [112], Bolis and Schilbach [113].

3.2 A Computational Model Describing the Experience of Social Agency

To formally describe the specific underlying mechanisms of social agency (SA; defined by μ_SA and σ²_SA; Figure 3), we put forward a multimodal hierarchical Bayesian model (Figure 3; cf. [113], [114], [115]). Here, hierarchical learning states (i. e. dynamic psychophysiological processes) and parameters (relatively stable processes, such as personal tendencies) describe the production and updating of social expectations and reactions. In the proposed scheme, social agency is modelled as a Gaussian probability distribution whose expected value (μ_SA) is controlled by the hierarchical mechanisms predicting two crucial processes, i. e., congruency (e. g. probability of gaze following) and contingency (e. g. latency of gaze following). While, belief updating is accomplished proportionally to precision weighted prediction-errors, the integration of the factors that models social agency is based on inverse-prediction-errors (IPE). Such a scheme captures two cardinal attributes of our hypothesis: First, it encapsulates a core aspect of predictive coding accounts, namely the tendency of an agent to update their beliefs slower when they are unsure about the validity of incoming information compared to prior beliefs (i. e. low relative precision).^[1] Secondly, based on our conceptual discussion above, the suggested scheme allows for modelling a strong sense of social agency when prediction errors are low (i. e., high IPE; in other words, strong sense of social agency reflects situations where expectations about the reactions of the other are met). Furthermore, the more confident an agent with this scheme is about their social expectations (e. g. congruency and contingency) the more confidence they place on their current state of social agency (i. e. lower σ²_SA). Experimentally, the experience of social agency could be measured by an explicit judgment or rating after varying interactive situations.

More concretely, the computational model deploys two families of hierarchical states, which track interpersonal congruency and contingency. Ascending the hierarchy, this scheme captures increasing temporal scales and levels of abstraction. Taking congruency of eye contact as an example, the first level represents one’s expectation on whether or not the other will make eye contact in an interaction. The second level captures the probability of this reaction to occur and the third level represents the volatility of such a probabilistic event. By extension, further hierarchical levels may capture deeper abstraction, e. g. volatility of volatility etc. Such processes are inherently interpersonal as they rely on attributes of oneself and the other. Furthermore, capturing varying temporal scales via such a hierarchical scheme is of crucial importance. To use an illustrative example, one could be in bad mood due to a conflict in one area of life. This might temporarily affect the expectations regarding the behaviour of others, e. g. the expectation of closer intimacy, which could lead to an interpersonal mismatch. The behaviour of another person might be mistaken as avoidance based on different expectations (e. g. frequency of eye contact). The degree of flexibility of such expectations is an important factor, as it might be what distinguishes some psychiatric conditions from temporary states of minds. In our predictive coding model, this is captured via the precision (π_x and π_y) of a state. The more confident one is about a belief the more precision one places on that belief. Finally, in this schema various predictive processes of congruency and contingency are integrated within a meta-Bayesian level in a precision-weighted and individualised fashion [114], [115]. Firstly, the more confident one is about the validity of a factor the more this factor contributes to a sense of social agency. Secondly, a person might rely more on specific cues than others. Importantly, such reliance is not taken as an exclusively biologically determined attribute, but rather as interplay between biology and sociocultural learning [21].

While this computational framework makes concrete predictions about the neural mechanisms of social agency, it should not be taken as strictly normative, as it allows for subject-specific fitting. Furthermore, this framework allows room for contrasting competing hypotheses about more specific details of these processes depending on the individual, the dyad and the social context. Here, it is important to clarify that the full model can be formally put to the test against alternative versions of different degrees of complexity by means of established Bayesian model selection (cf. [116]). Reduced or extended models both in terms of the number and type of modalities (e. g. gaze or other social cues) and number of levels can be considered. Importantly, this scheme can be also extended for accommodating multimodal hierarchical processes within social interactions [113]. Taken together, we suggest that the probability to experience social agency can be described by predictions about the outcomes of social encounters and the contingency and congruency of the social cues that are perceived during the interaction. A mismatch between the prior expectation of social behaviour and the actual sensory consequence affects social agency and updates these predictions. It can be hypothesised that the flexibility of this system (meaning the extent to which the predictions are updated) is dependent on different tendencies or abilities of the individual as well as other factors within the environment (Figure 2). By using simulations with virtual realities, systematically manipulating the contingency as well as congruency of interactive agents and applying the proposed computational approach, it is possible to study the underlying mechanisms of social agency. Of course, the verification of the here mentioned hypothesis remains open to future empirical data, motivating concrete research avenues based on interactive and naturalistic but also systematically controlled artificial systems. Furthermore, it has been suggested that different psychiatric disorders might reflect different styles of updating expectations about the (social) world and generating interaction patterns [20], [117], [118]. Consequently, fitting a model such as the one suggested above to different groups of individuals (e. g. diagnosed with a psychiatric condition) will allow studying condition-specific differences in social agency formation processes on both the behavioural and neural level.