Abstract
This present article is part of a larger study on speaker-hearer allocation of attentional resources in face-to-face interactions. The goal of the paper is twofold: first, we present results concerning the degree of correlation, in computer-mediated conversation, between speaker’s timing and intensity of smiling when humor is either present or absent in the conversation. The results were obtained from the analysis of five dyadic interactions between English speakers that were video and audio recorded, transcribed, and analyzed to establish a baseline of synchronicity of smiling among participants. From the study it emerged that conversational partners engaged in humorous conversations not only reciprocate each other’s smiling, but also match each other’s smiling intensity. Our data led to the identification of different smiling and non-smiling synchronic behaviors that point to the existence of a synchronous multimodal relationship between humorous events and smiling intensity for conversational partners. Second, in the last part of the paper, we argue for the need of a multimodal conversational corpus in humor studies and present the corpus that is being collected, annotated, and analyzed at Texas A&M University–Commerce. The corpus consists of humorous interactions among dyads of native speakers of English, Spanish, and Chinese for which video, audio, and eye-tracking data have been recorded. As part of this section of the paper, we also present some preliminary results based on the analysis of one English conversation, and some exploratory analysis of Chinese data, that show that greater attention is paid to facial areas involved in smiling when humor is present. This study sheds light on the role of smiling as a discourse marker (Attardo, S., L. Pickering, F. Lomotey & S. Menjo. 2013. Multimodality in Conversational Humor. Review of Cognitive Linguistics 11(2). 400–414.), and therefore as a meaningful device in verbal communication.
About the authors
Elisa Gironzetti received her PhD in Spanish Language Teaching from the University of Alicante (Spain) with a dissertation on humorous and non-humorous communication through single-panel cartoons. She teaches in the Spanish undergraduate and graduate programs at Texas A&M University–Commerce. Her research focuses on humor and Spanish language teaching, and she is currently performing eye-tracking experiments to study the interaction of humor, gaze, and smiling in conversation.
Lucy Pickering is Associate Professor and director of the Applied Linguistics Laboratory at Texas A&M University–Commerce. She received her Ph.D. in Applied Linguistics in 1999 from the University of Florida and has taught in University of Alabama, Georgetown University, and Georgia State University. Her research program is focused on spoken discourse. She has done considerable work with Brazil’s model of Discourse Intonation and its application to second language classroom discourse.
Meichan Huang received her M.Sc. in TESOL (Teaching English to the Speakers of Other Languages) from the University of Edinburgh, and B.A. in English Translation from Sichuan International Studies University in China. Her major research interest is humor in second language classrooms, with a focus on the use of verbal humor as teaching material. She is also interested in second language acquisition and language pedagogy.
Ying Zhang holds a PhD in English Linguistics and Literature from Shanghai International Studies University. She is now a lecturer in the School of Foreign Languages at Shanghai University. She worked as a visiting scholar sponsored by China Scholarship Council in the Department of Literature and Languages at Texas A&M University-Commerce from April 2014 to April 2015. Her research programs focus on conversational humor. Her major research interests are humor, pragmatics, sociolinguistics and applied linguistics.
Shigehito Menjo received his M.A in Japanese Language and Pedagogy, and B.A in Linguistics from University of Oregon. He also received a TESOL certificate from Texas A&M University–Commerce. He has several years of experience in teaching Japanese, first year composition, and TESOL methods classes at universities in the United States. His major research interest is the acquisition of prosody in second language and the use of smiling and prosody in the spoken discourse of humor/comedy.
Salvatore Attardo is Dean of the College of Humanities, Social Sciences and Arts at Texas A&M University-Commerce, where he is also Full Professor of Linguistics. He hold degrees from The Catholic University of Milan (1986) and Purdue University (PhD 1991). He has published extensively in humor, pragmatics, and semantics, primarily on issues relating to implicatures, irony, rationality and more generally on Neo-Gricean Pragmatics. His other areas of interest are in sociolinguistics, cognitive linguistics, and computational semantics.
References
Anolli, L. & L. Lambiase. 1990. “Giochi di sguardo” nella conversazione. Giornale Italiano di psicologia 17(1). 27–58.Search in Google Scholar
Attardo, S. 2001. Humorous texts. Berlin: Mouton de Gruyter.10.1515/9783110887969Search in Google Scholar
Attardo, S. 2012. Smiling, laughter, and humor. In Paolo Santangelo (ed.), Laughing in Chinese, 421–436. Rome: Aracne.Search in Google Scholar
Attardo, S, L. Pickering & A. Baker. 2011. Prosodic and multimodal markers of humor in conversation. Pragmatics and Cognition 19(2). 224–247.10.1075/bct.55.03attSearch in Google Scholar
Attardo, S., L. Pickering, F. Lomotey & S. Menjo. 2013. Multimodality in conversational humor. Review of Cognitive Linguistics 11(2). 400–414.10.1075/bct.78.12attSearch in Google Scholar
Beattie, G. W. 1978. Floor apportionment and gaze in conversational dyads. British Journal of Social and Clinical Psychology 17(1). 7–15.10.1111/j.2044-8260.1978.tb00889.xSearch in Google Scholar
Brône, G. & B. Oben. 2015. InSight Interaction: A multimodal and multifocal dialogue corpus. Language Resources and Evaluation 49(1). 195–214.10.1007/s10579-014-9283-2Search in Google Scholar
Broz, F., H. Lehmann, C. L. Nehaniv & K. Dautenhahn. 2012. Mutual gaze, personality, and familiarity: Dual eye-tracking during conversation. IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, 858–864.10.1109/ROMAN.2012.6343859Search in Google Scholar
Calvo, Manuel G., Andrés Fernández-Martín & Lauri Nummenmaa. 2013a. A smile biases the recognition of eye expressions: Configural projection from a salient mouth. The Quarterly Journal of Experimental Psychology 66(6). 1159–1181, DOI: 10.1080/17470218.2012.732586.Search in Google Scholar
Calvo, Manuel G., Aida Gutiérrez-García, Pedro Avero & Daniel Lundqvist. 2013b. Attentional mechanisms in judging genuine and fake smiles: Eye-movement patterns. Emotion 13(4). 792–802. DOI: 10.1037/a0032317.Search in Google Scholar
Cappella, J. N. 1997. Behavioral and judged coordination in adult informal social interactions: Vocal and Kinesic Indicators. Journal of Personality and Social Psychology 72(1). 119–131. http://dx.doi.org/10.1037/0022-3514.72.1.119.10.1037/0022-3514.72.1.119Search in Google Scholar
Cary, M. 1978. The role of gaze in the initiation of conversation. Social Psychology 41. 269–271.10.2307/3033565Search in Google Scholar
Chafe, W. 1994. Discourse, consciousness, and time. The flow and displacement of conscious experience in speaking and writing. Chicago: University of Chicago Press.Search in Google Scholar
Chartrand, T. L. & J. A. Bargh. 1999. The chameleon effect: The perception-behavior link and social interaction. Journal of Personality and Social Psychology 76(6). 893–910. doi:10.1037/0022-3514.76.6.893.Search in Google Scholar
Dale, R., R. Fusaroli, N. D. Duran & D. C. Richardson. 2013. The self-organization of human interaction. Psychology of Learning and Motivation 59. 43–96.10.1016/B978-0-12-407187-2.00002-2Search in Google Scholar
Dale, R., R. Fusaroli, D. D. Håkonsson, P. Healey, D. Mønster, J. J. McGraw, P. Mitkidis & K. Tylén. 2013. Beyond synchrony: Complementarity and asynchrony in joint action. Proceedings of CogSci 2013.Search in Google Scholar
Delaherche, E. & M. Chetouani. 2010. Multimodal Coordination: Exploring Relevant Features and Measures. In Proceedings of ACM Multimedia Second International Workshop. Social Signal Processing. Florence, Italy.10.1145/1878116.1878131Search in Google Scholar
Ekman, P. & W. V. Friesen. 1978. Facial action coding system. Palo Alto, California: Consulting Psychologists Press.10.1037/t27734-000Search in Google Scholar
Ekman, P., E. R. Sorenson & W. V. Friesen. 1969. Pan-cultural elements in facial displays of emotion. Science 164. 86–88.10.1126/science.164.3875.86Search in Google Scholar
Elfenbein, H. A. & N. Ambady. 2002. On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychology Bulletin 128. 203–235.10.1037/0033-2909.128.2.203Search in Google Scholar
Fernández-Martín, A. & M. Gutiérrez-Calvo. 2012. Curso temporal en la discriminación de la sonrisa: medidas de latencia sacádica. Psicológica 33. 319–343.Search in Google Scholar
Fernández-Martín, A., A. Gutiérrez-García & M. G. Calvo. 2013. A smile radiates outwards and biases the eye expression. Spanish Journal of Psychology 16(53). 1–11.10.1017/sjp.2013.68Search in Google Scholar
Fusaroli, R., J. Rączaszek-Leonardi & K. Tylén. 2014. Dialog as interpersonal synergy. New Ideas in Psychology 32. 147–157. doi:10.1016/j.newideapsych.2013.03.005.Search in Google Scholar
Fusaroli, R. & K. Tylén. 2012. Carving language for social interaction: a dynamic approach. Interaction Studies 13(1). 103–123.10.1075/is.13.1.07fusSearch in Google Scholar
Gervais, M. & D. S. Wilson. 2005. The evolution and functions of laughter and humor: A synthetic approach. Quarterly Review of Biology 80(4). 395–430.10.1086/498281Search in Google Scholar
Gironzetti, E., S. Attardo & L. Pickering. 2016. Smiling, gaze, and humor in conversation: A Pilot Study. In L. Ruiz Gurillo (ed.), Metapragmatics of humor: Current research trends. Amsterdam: John Benjamins.10.1075/ivitra.14.12girSearch in Google Scholar
Gironzetti, E. & S. Menjo. 2014. Smiling as a Discourse Marker of Humor. Paper presented at the 2nd Conference of the American Pragmatics Association (AMPRA), October 17–19, 2014. University of California, Los Angeles.Search in Google Scholar
Hay, J. 2001. The pragmatics of humor support: Humor – International Journal of Humor Research. Humor: International Journal of Humor Research 14(1). 55–82. http://doi.org/10.1515/humr.14.1.55.http://doi.org/10.1515/humr.14.1.55Search in Google Scholar
Heerey, E. A. & H. M. Crossley. 2013. Predictive and reactive mechanisms in smile reciprocity. Psychological Science 24(8). 1446–1455.10.1177/0956797612472203Search in Google Scholar
Heerey E. A. & A. M. Kring. 2007. Interpersonal consequences of social anxiety. Journal of Abnormal Psychology 116. 125–134.10.1037/0021-843X.116.1.125Search in Google Scholar
Hess U. & P. Bourgeois. 2010. You smile – I smile: Emotion expression in social interaction. Biological Psychology 84. 514–520.10.1016/j.biopsycho.2009.11.001Search in Google Scholar
Holt, E. & P. J. Glenn. 2013. Studies of Laughter in Interaction London, UK: Bloomsbury AcademicSearch in Google Scholar
Jefferson, G. 1979. “A technique for inviting laughter and its subsequent acceptance declination.” In George Psathas (ed.), Everyday language: Studies in ethnomethodology, 79–96. New York: Irvington.Search in Google Scholar
Kendon, A. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26. 22–63.10.1016/0001-6918(67)90005-4Search in Google Scholar
Kleinke, C. L. 1986. Gaze and eye contact: A research review. PsyChological Bulletin 100(1). 78–100.10.1037/0033-2909.100.1.78Search in Google Scholar
Knackstedt, G. & C. Kleinke. 1991. Eye contact, gender, and personality judgments. The Journal of Social Psychology, 131. 303–304.10.1080/00224545.1991.9713855Search in Google Scholar
Lakin, J. L. & T. L. Chartrand. 2003. Using nonconscious behavioral mimicry to create affiliation and rapport. Psychological Science 14(4). 334–339. doi:10.1111/1467-9280.14481.Search in Google Scholar
Levine, M. H. & P. Sutton-Smith. 1973. Effects of age, sex and task on visual behavior during dyadic interaction. Developmental Psychology 9. 400–405.10.1037/h0034929Search in Google Scholar
Lumsden, J., K. M. Lynden, M. J. Richardson, C. A. Smith & C. N. Macrae. 2012. “Who Syncs? Social motives and interpersonal coordination.” Journal of Experimental Social Psychology 48(3). 746–751. Retrieved December 2, 2014 (http://www.sciencedirect.com/science/article/pii/S0022103111002976).10.1016/j.jesp.2011.12.007Search in Google Scholar
Macrae, C. N., O. K. Duffy, L. K. Miles & J. Lawrence. 2008. A case of hand waving: Action synchrony and person perception. Cognition 109. 152–156.10.1016/j.cognition.2008.07.007Search in Google Scholar
Miles, L. K., J. L. Griffiths, M. J. Richardson & C. N. Macrae. 2010. Too late to coordinate: Contextual influences on behavioral synchrony. European Journal of Social Psychology 40. 52–60.10.1002/ejsp.721Search in Google Scholar
Miles, L. K., L. K. Nind & C. N. Macrae. 2009. The rhythm of rapport: Interpersonal synchrony and social perception. Journal of Experimental Social Psychology 45(3). 585–589. http://doi.org/10.1016/j.jesp.2009.02.002.10.1016/j.jesp.2009.02.002Search in Google Scholar
Norrick, N. R. 1993. Conversational joking: Humor in everyday talk. Bloomington, Indiana: Indiana University Press.Search in Google Scholar
O‘Donnell-Trujillo, N. & K. Adams. 1983. Heheh in conversation: Some coordinating accomplishments of laughter. Western Journal of Speech Communication 47. 175–191.10.1080/10570318309374114Search in Google Scholar
Paxton, A. & R. Dale. 2013. Multimodal networks for interpersonal interaction and conversational contexts. Proceedings of the 35th Annual Meeting of the Cognitive Science Society. Retrieved December 2, 2014 (http://alexandrapaxton.com/wp-content/uploads/2013/05/Paxton-Dale-2013-CogSci.pdf).Search in Google Scholar
Pickering, L., M. Corduas, J. Eisterhold, B. Seifried, A. Eggleston & S. Attardo. 2009. Prosodic markers of saliency in humorous narratives. Discourse Processes 46. 517–540.10.1080/01638530902959604Search in Google Scholar
Provine, R. R. 2004. Laughing, tickling, and the evolution of speech and self. Current Directions in Psychological Science 13(6). 215–218. http://doi.org/10.1111/j.0963-7214.2004.00311.x10.1111/j.0963-7214.2004.00311.xSearch in Google Scholar
Raidt, S., G. Bailly & F. Elisei. 2007. Analyzing and modeling gaze during face-to-face interactions. 7th International Conference on Intelligent Virtual Agents (IVA), Paris, France, 100–101.Search in Google Scholar
Raskin, V. 1985. Semantic mechanisms of humor. Dodrecht, the Netherlands: D. Reidel.10.1007/978-94-009-6472-3Search in Google Scholar
Rosengrant, D., D. Hearrington, K. Alvarado, & D. Keeble. 2012. Following Student Gaze Patterns in Physical Science Lectures. AIP Conference Proceedings, 1413. 323–326. doi: http://dx.doi.org/10.1063/1.368006010.1063/1.3680060Search in Google Scholar
Sacks, H. 1974 [1989]. An analysis of the course of a joke’s telling in conversation. In Bauman, R. & Joel Sherzer (eds.) Explorations in the ethnography of speaking, 2nd edn, 337–353. Cambridge: Cambridge University Press.10.1017/CBO9780511611810.022Search in Google Scholar
Sauter, D. A., F. Eisner, P. Ekman & S. K. Scott. 2010. Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations. Proceedings of the National Academy of Sciences 107(6). 2408–2412. http://doi.org/10.1073/pnas.0908239106.http://doi.org/10.1073/pnas.0908239106Search in Google Scholar
Schegloff, E. A., G. Jefferson & H. Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Language 53. 361–382.10.1353/lan.1977.0041Search in Google Scholar
Scott, S. K., N. Lavan, S. Chen & C. McGettigan. 2014. The social life of laughter. Trends in Cognitive Sciences 18(12). 618–620. http://doi.org/10.1016/j.tics.2014.09.002.http://doi.org/10.1016/j.tics.2014.09.002Search in Google Scholar
Tschacher, W., G. M. Rees & F. Ramseyer. 2014. Nonverbal synchrony and affect in dyadic interactions. Frontiers in Psychology 5. 1323. doi:10.3389/fpsyg.2014.01323.Search in Google Scholar
Valdesolo, P., J. Ouyang & D. A. DeSteno. 2010. The rhythm of joint action: Synchrony promotes cooperative ability. Journal of Experimental Social Psychology 46. 693–695.10.1016/j.jesp.2010.03.004Search in Google Scholar
van Hooff, J. 1972. A comparative approach to the phylogeny of laughter and smiling. In R. A. Hinde (ed.), Nonverbal communication. Cambridge: Cambridge University Press.Search in Google Scholar
Vertegaal, R., R. Slagter, G. van der Veer & A. Nijholt. 2001. Eye gaze patterns in conversation: There is more to conversational agents than meets the eyes. CHI 3(1). 301–308.10.1145/365024.365119Search in Google Scholar
Wild B., M. Erb, M. Eyb, M. Bartels & W. Grodd. 2003. Why are smiles contagious? An fMRI study of the interaction between perception of facial affect and facial movements. Psychiatry Research 123. 17–36.10.1016/S0925-4927(03)00006-4Search in Google Scholar
Williams, J. A., E. L. Burns & E. A. Harmon. 2009. Insincere utterances and gaze: Eye contact during sarcastic statements. Perceptual and Motor Skills 108. 565–572.10.2466/pms.108.2.565-572Search in Google Scholar
Appendix A. Smiling Intensity Scale
The five levels of this Smiling Intensity Scale (SIS) are descriptive of different smiling behaviors:
Level 0: Neutral. No smile, no flexing of the zygomaticus (no AU12), may show dimpling (AU14) or squinting of the eyes (caused by AU6 or AU7), but no raised side of the mouth (no AU 12), the mouth may be closed or open (AU25 or AU26).
Level 1: Closed mouth smile. Shows flexing of the zygomaticus (AU12), may show dimpling (AU14) and may show flexing of the orbicularis oculi (caused by AU6 or AU7).
Level 2: Open mouth smile. Showing upper teeth (AU25), flexing of the zygomaticus (AU12), may show dimpling (AU14), may show flexing of the orbicularis oculi (caused by AU6 or AU7).
Level 3: Wide open mouth smile. Shows flexing of the zygomaticus (AU12), flexing of the orbicularis oculi (caused by AU6 or AU7), and may show dimpling (AU14). 3A: showing lower and upper teeth (AU25), or 3B: showing a gap between upper and lower teeth (AU25 and AU26).
Level 4: Laughing smile. The jaw is dropped (AU25 and AU26 or AU27), showing lower and upper teeth, flexing zygomaticus (AU12), flexing of the orbicularis oculi (AU6 or AU7), dimpling (AU14)
©2016 by De Gruyter Mouton