Non-convergent boundaries and action ascription in multimodal interaction

3.2.1 Incipient speakership

In Example 1, the participants, who we nickname L for left and R for right, are eating a mince pie, which is a Christmas pastry filled with sticky dried fruit and spices. Although there are several “mms” in this extract, they are only shown to contextualise the relevant line. Our focus is the “mm” in line 16 as it marks incipient speakership: in saying “mm”, L projects a turn.

Lines 01–14 of the transcript show L and R discussing what hot beverages would go well with the mince pie that they are currently eating. This is followed by a long verbal lapse with a series of kinesic actions (line 15). Right before the end of the lapse, L begins pointing at the pie and covers her mouth with her other hand (Figure 1a). The lapse ends when L makes an “mm” token in line 16, while pointing to, and gazing at, the pie (Figure 1b). An obvious candidate action for “mm” in the context of tasting is displaying pleasure in eating, i.e., doing a gustatory noise. In our data gustatory, “mms” are not associated with hand gestures, but they do tend to co-occur with facial expressions such as frowns, smiles, and closed eyes (61% of the cases). However, neither L’s gaze nor her pointing behaviour is consistent with this analysis. Instead, L is securing her next turn, i.e., projecting upcoming talk, while still eating. Of the many ways in which people can audibly display incipient speakership, some, such as clicks, are incompatible with eating. On the other hand, “mm” is compatible with eating. Prosodic features of “mm” can also be manipulated without interfering with eating. L’s “mm” at line 16 starts with a glottal stop, is low in her pitch range, relatively short in duration (276 ms), and has a falling intonation contour (Figure 2). In the short silence after “mm”, L swallows, while maintaining her gaze and pointing gesture.

Figure 1

Screenshots (a, b, c, d, e and f) illustrating the embodied behaviour at lines 15, 16, 18, 22, 23, and 24, respectively.

Figure 2

Waveform (upper panel), spectrogram (middle), and F0/intensity trace of L’s “mm” (Example 1 line 16). F0 trace scaled to L’s minimum/maximum F0, median marked at 219 Hz.

While there are clear phonetic cesuras (glottalisation, pausing) at the start and end of the spoken token “mm”, L’s bodily behaviour after “mm” projects her upcoming action and selects the focus of that action. L maintains her gaze to the pie. These held non-verbal articulations signal that whatever was started during “mm” is not yet finished, despite a clear phonetic cesura at the end of the particle. The actions of looking and pointing, held after the production of the spoken “mm”, constitute it as a preface to talk, and not as something designed to be complete in itself. In other words, the apparently obvious phonetic cesura (break) at the end of “mm” is overridden as a possible TRP by the co-occurring bodily actions.

Indeed, in lines 18–19, L continues by asking a question. Also, at the beginning of L’s same turn, R shifts his gaze to L (Figure 1c), indicating his recipiency. The second part of L’s question in line 19 is a complete sentence by itself, “what does it have”. For this part of the turn, L shifts her gaze to R, selecting him as the next speaker and anticipating his next turn. Her pointing is still towards the pie, which makes it clear that the pie is the referent of the question. R shifts his gaze to the pie.

There is a long silence in the place where an answer is due (line 20); L’s gaze and gesture are held in this slot, so L is leaving the turn space open to R. This format projects a dispreferred response, or some kind of trouble in answering. R replies in line 21, with a non-answer response (Stivers and Robinson 2006) that claims no access and therefore grounds to reply, which could explain the longer silence in line 20. At this point, the adjacency pair could be closed, but there is a silence (line 22) and L maintains her gaze to R and her pointing gesture to the pie, although in a more relaxed manner (Figure 1d). By holding these steady, she continues to indicate that the turn space is R’s and treats the answer “I don’t know” as inadequate, and by withholding her own talk, she leaves space for him to offer another response or an account.

Then, R provides another candidate answer in line 23, “pastry”. He produces post-completion laughter particles, so marking his answer as not serious – in itself an orientation to his understanding of the question being about the contents of the pie, not its shell: it is self-evident that a pie has a shell, i.e., pastry. When he starts his laughter, he turns his gaze to L, an invitation for her to join in with his laughter (Figure 1e; Jefferson 1979). At this point and as a consequence, L releases her pointing to the pie and gazes away from it. Only then, she starts a new gesture (Figure 1f) and a next sequence of talk in which she recycles her question about the filling of the pie. Figure 3 shows the analysed extract and the visible behaviour more schematically.

Figure 3

R’s gaze behaviour (annotated left), alongside L’s gustatory, manual, and gaze behaviour (annotated right).

Marking incipient speakership with “mm” can be ascribed from other visible behaviours at line 16, namely, L’s gaze and pointing, which are held, and the fact that she is visibly eating/swallowing. By gazing and pointing at the pie while eating, and by saying “mm” at the same time, L is able to project not just incipient talk, but to indicate something about what her talk will be about; her point and gaze at the pie handle reference; her gaze to R during the adjacency pair secure R’s recipiency; and her verbal and non-verbal behaviour at the end of the sequence contribute to closing the sequence off.

Example 2 is similar to Example 1 as it shows another instance of “mm” marking incipient speakership in which the boundaries of the speech and the other embodied conduct do not align. The speaker who produces “mm”, L, is visibly eating and chewing as well. However, as we will see, the embodied behaviour is not the same, but the recipient is able to ascribe an action to L’s observable vocal and gestural behaviour in the same fashion. In Example 2, there is no pointing, but there is a head tilt, gazing at, and holding of the food for closer inspection. In both examples, eye gaze from the co-participant is secured.

Before the beginning of the excerpt shown for Example 2, R begins to taste the chocolate as L begins to smell it and produces a description of that in line 01. Responsive to this, R brings the remaining piece of chocolate he has just started eating closer to his nose, smells it, and agrees with the minimal acknowledgement token “mm hm” in line 03. Then, R produces a description of the taste in line 05, recycling L’s description from line 01 and foregrounding the verb sabe “it tastes”, and gazes at L in the middle of the turn.

Similar to what happens in Example 1, even when there are phonetic cesuras around “mm”, some of the physical gestures are indexing the referent about which something is to be said and the boundaries for the embodied behaviour do not coincide with the verbal ones. A total of 0.2 s before “mm” is produced, L shifts his gaze to the piece of chocolate he is holding. As L begins to produce “mm”, he tilts his head to the side (Figure 4a) and begins twisting his fingers as if to inspect the food (an action that is maintained until the end of line 07).

Figure 4

Screenshots (a and b) illustrating the embodied behaviour at line 7.

As L is tasting the chocolate, he produces the “mm” in line 07 marking incipient speakership as his mouth is engaged with food and as already mentioned, and the nasal manner of articulation of “mm” is compatible with eating. L’s “mm” at line 07 starts with a glottal stop, is low in his pitch range, is short in duration (123 ms), and has a falling intonation contour (Figure 5). These features are representative of our findings for incipiency (Table 4). The majority of the non-response and response tokens marking incipient speakership have a falling intonation, half of them have glottalisation at the beginning, and two-thirds have accompanying head movements.

Figure 5

Waveform (upper panel), spectrogram (middle), and F0/intensity trace of L’s “mm” (Example 2 line 15). F0 trace scaled to L’s minimum/maximum F0, median marked at 133 Hz.

At this point, R is gazing at his own piece of chocolate, but right after “mm” is produced, and he turns his gaze towards L who is visibly chewing and swallowing (Figure 4b). In this example, “mm” secures eye gaze and allows the recipient to see the speaker is still engaged with eating but has manifested their gearing up to speak. In half of the examples of incipient speakership, the recipient turns their gaze to the speaker when “mm” is produced. Another piece of evidence that supports the claim that the "mm" in line 07 is projecting incipient speakership, is the silence after it. This silence takes 0.7 s in which R maintains his gaze towards L and does not take a turn orienting to the turn rights claimed by L. The silence is another phonetic cesura at the end of the particle; however, the held non-verbal articulations project upcoming talk (cf. Sikveland and Ogden 2012). After the silence, L produces a first assessment está exquisito “it’s exquisite” in line 07. After a 0.3 silence, R produces me cargó “I loathed it”, a second assessment that is formatted as a first in line 09.

As can be seen, the physical gestures align with other boundaries which makes “mm” hearable as projecting more talk. Even if the physical gestures sometimes coincide with beginnings or endings of verbal material, this is better explained by the intertwining of the parallel activities that the interactants are involved in.

3.2.2 Gustatory “mms”

We saw that when “mm” marks incipient speakership, it occurs more commonly in non-response sequential position, but it can also occur as a response token. It tends to have glottalisation, a falling intonation contour, and be short in duration. The co-occurring embodied behaviour (visibly eating, pointing, and gazing at the food) contributed to its action ascription. Gustatory “mm” tokens, on the other hand, only occur in non-response position in our data. They tend to have a rise­falling intonation contour that matches what Gardner (2001) considers a characteristic prosodic shape for the gustatory “mm” in English. As for the co-occurring visible behaviour, there is more variety, but as we will see, its sequential placement plays an important role for action ascription.

Example 3 shows an instance of “mm” whose design features point towards its understanding as gustatory. In the 7.9 s gap in line 01 of Example 3, both interactants try baked beans at the same time while looking ahead and not engaging in mutual gaze.

While chewing, R gazes at L (Figure 6) who is still gazing away, and R produces “mm” in line 02. This “mm” is long (780 ms) and has a rise-falling intonation contour from a high pitch onset (Figure 7). R produces this “mm” smiling and with his eye gaze directed towards L (both behaviours began 0.8 s before “mm” is produced). The facial expression, a smile, conveys the speaker’s positive stance towards the food he is eating. The eye gaze is used to mobilise a response (cf. Stivers and Rossano 2010; Rossano 2012), i.e., find out what L thinks. Our data show that gustatory “mms” are accompanied by speaker gaze towards the recipient in only 22% of the cases. However, eye gaze has been shown to mobilise actions, and in Example 3, R gazes at L to mobilise a first assessment while positioning himself in second position to assess (cf. González Temer 2017 on food assessments).

Figure 6

Screenshot illustrating the embodied behaviour at line 2.

Figure 7

Waveform (upper panel), spectrogram (middle), and F0/intensity trace of R’s “mm” (Example 3, line 02). F0 trace scaled to L’s minimum/maximum F0, median marked at 141 Hz.

As L is finishing swallowing in line 03, he raises his right eyebrow, slightly tilts his head to one side, moves his right arm, and then starts talking. This suggests that he treats R’s gaze towards him as mobilising a response from him (cf. González Temer 2017; Stivers and Rossano 2010 on eye gaze in mobilising an assessment). As L starts producing his assessment in line 04, R looks away and L gazes at R. Up to that point, L has only assessed the food as agridulce, “sweet and sour” which does not have any positive or negative value per se. R produces a positive assessment in line 06, which was already projected by his gustatory “mm” token in line 02. R’s positive assessment is followed by a weak agreement at line 07 with a quiet sí “yes”. In this example, both the facial expression and eye gaze shift begin slightly before “mm” is uttered, which also means that the phonetic cesuras around “mm” do not align with those of the embodied action. Nevertheless, the combination of these features at a point of verbal and nonverbal convergence makes this token multipurpose, i.e., displaying a stance about the experience of eating and mobilising an assessment.

The next example (4) shows another instance of a gustatory token in non-response position, projecting a positive assessment. As in example 3, this “mm” is also long and has a rise-fall intonation contour which is common for this type of token as was previously shown in Table 4. Gustatory tokens are claimed to be done in response to a non-talk stimulus (pleasure in eating or the prospect of it; cf. Gardner 2001; Wiggins 2002). At the same time, these tokens are inevitably understood as projecting a positive stance about the food.

Before the excerpt shown, L tastes the mince pie first and then R tastes it. In line 01, we see the 6.6 s that represents silent eating time. Then, L takes a napkin to wipe his mouth, signalling that he is finishing eating. This also signals that his mouth is not engaged with food anymore, and therefore, he is available to speak, which in this context makes relevant an assessment as a next action. While still chewing, he produces “mm” in line 02 and then he wipes his lips with the napkin (Figure 8a). The token has a rise-fall intonation contour, a duration of 567 ms and ends mid in the speaker’s pitch range (Figure 9).

Figure 8

Screenshots (a and b) illustrating the embodied behaviour at lines 2 and 4.

Figure 9

Waveform (upper panel), spectrogram (middle), and F0/intensity trace of R’s “mm” (Example 4, line 02). F0 trace scaled to R’s minimum/maximum F0, median marked at 117 Hz.

In line 04, L, while smiling, asks R whether he liked the food, which orients to “mm” as a display of enjoyment and at the same time displays her affiliation with the stance and the understanding that R is ready to make an assessment (Figure 8b). The next turn at line 08 is the beginning of a sequence which is an attempt by both speakers to find common ground on which to assess (the work of Liberman (2013) on coffee tasting), by comparing what they have just eaten with other foods they are familiar with. In other words, “mm” is placed at a point in the ongoing activity where an assessment is both a relevant and possible next action, and L and R treat it as a preface to this activity.

In Example 3, we saw how the combination of this type of “mm” together with eye gaze directed to the interlocutor and smiling also projected a positive stance, but in addition, it mobilised a first assessment. In Example 4, there is a neutral facial expression, and the eye gaze is directed toward the napkin that the speaker has just grabbed, so while these non-verbal features might not be contributing to a projection of a stance, the place in the interaction in which the token occurs helps to interpret the token in situ as both interactants have been eating silently for over 7 s.

In sum, the different actions that “mm” performs are accompanied by different verbal and physical forms, in different sequential positions. When “mm” marks incipient speakership, we found that it was possible to have all the intonation contours of Chilean Spanish except rise-fall. There seem to be no particular constraints on the speaker’s facial expression. Conversely, gustatory “mm” tokens – that is, ones that express some kind of response to eating and project some verbal assessment of the food – can bear the full range of tones of the language and are accompanied by some facial expression that is convergent with the vocal channel. However, these exponents of these actions which are done through “mm” do not always have convergent boundaries, and so they may extend over divergent temporal domains; this property is a resource for meaning and part of the richness of the semantic underspecification of “mm”.

The examples in Section 3 have illustrated that several actions are ascribable to “mm” such as marking incipient speakership or displaying pleasure in eating as a gustatory token. These actions are made available through aspects of phonetic, prosodic, and non-vocal design, as well as sequential positioning. We see that the boundaries of the linguistic (including prosodic/phonetic) units and bodily behaviours such as gestures, gaze behaviour, and facial expression do not coincide with one another. These non-aligned boundaries help create windows of opportunity – moments in time – for interactants to do different actions with what at first sight seems to be the same token. In the following section, we provide a more generic account of this type of particle by drawing on the idea of composite utterance in the spirit of work by Clark (1996) or Enfield (2009).