Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter Mouton September 21, 2020

“Do you understand (me)?” negotiating mutual understanding by using gaze and environmentally coupled gestures between two deaf signing participants

  • Nina Sivunen EMAIL logo and Elina Tapio


In this paper we explore the use of multimodal and multilingual semiotic resources in interactions between two deaf signing participants, a researcher and an asylum seeker. The focus is on the use of gaze and environmentally coupled gestures. Drawing on multimodal analysis and linguistic ethnography, we demonstrate how gaze and environmentally coupled gestures are effective semiotic resources for reaching mutual understanding. The study provides insight into the challenges and opportunities (deaf) asylum seekers, researchers, and employees of reception centres or the state may encounter because of the asymmetrical language competencies. Our concern is that such asymmetrical situations may be created and maintained by ignoring visual and embodied resources in interaction and, in the case of deaf asylum seekers, by unrealistic expectations towards conventionalized forms of international sign.

Corresponding author: Nina Sivunen, University of Jyväskylä, Centre of Applied Language Studies, Jyväskylä, Finland, E-mail:


We want to express our thanks to Sari Pöyhönen, Mia Halonen, Lindsay Ferrara and Teppo Jakonen for their valuable comments and insights on previous versions of this article. We would also like to thank the anonymous reviewers whose detailed comments and suggestions helped to improve the manuscript. Nina Sivunen is grateful for the support from the Centre of Applied Language Studies (CALS) of the University of Jyväskylä.

  1. Research funding: This research was funded by the University of Jyväskylä, Faculty of Humanities and Social Sciences, Humak University of Applied Sciences and by the EU Home Affairs Funds (SMDno:2016-1561) through the PICCORE project.



beginning of overlapping sign


end of overlapping sign


pauses timed in tenths of a second (approximately)


gloss for a manual sign and a gesture


constructed dialogue


pointing to another interlocutor


lexical pointing direction


pointing to an object in the environment






head shakings


head noddings

shdr dwn

shoulders down


eyegaze direction, head and body activities, non-manual elements (e.g. raised eyebrows)

fig. x

The location of the picture (figure x) in the partiture


Bateman, John, Janina Wildfeuer & Tuomo Hiippala. 2017. Multimodality: Foundations, research and analysis a problem-oriented introduction. Berlin: De Gruyter.10.1515/9783110479898Search in Google Scholar

Bezemer, Jeff & Gunther Kress. 2014. Touch: A resource for making meaning. Australian Journal of Language and Literacy 37(2). 77–85.Search in Google Scholar

Blommaert, Jan, James Collins & Stef Slembrouck. 2005. Spaces of multilingualism. Multilingualism and Diasporic Populations: Special Issue of Language and Communication. 25(2). 197–215. in Google Scholar

Copland, Fiona & Angela Creese. 2015. Linguistic ethnography: Collecting, analysing and presenting data. London: Sage.10.4135/9781473910607Search in Google Scholar

Cormier, Kearsy, Sandra Smith & Zed Sevcikova-Sehyr. 2015. Rethinking constructed action. Sign Language and Linguistics 18(2). 167–204. in Google Scholar

Crasborn, Onno & Anja Hiddinga. 2015. The paradox of international sign: The importance of deaf-hearing encounters for deaf-deaf communication across sign language borders. In Michele Friedner & Annelies Kusters (eds.), It’s a Small World: International deaf spaces and encounters, 59–69. Washington, DC: Gallaudet University Press.10.2307/j.ctv2rcnnjs.9Search in Google Scholar

Crasborn, Onno & Han Sloetjes. 2008. Enhanced ELAN functionality for sign language corpora. In: Proceedings of LREC 2008, sixth international conference on language resources and evaluation.Search in Google Scholar

Drew, Paul. 1997. ‘Open’ class repair initiators in response to sequential sources of troubles in conversation. Journal of Pragmatics 28(1). 69–101. in Google Scholar

Engberg-Pedersen, Elisabeth. 2013. From pointing to reference and predication: Pointing signs, eyegaze, and head and body orientation in Danish sign language. In Sotaro Kita (ed.), Pointing: Where language, culture, and cognition meet, 269–292. Mahwah, NJ: Erlbaum.Search in Google Scholar

Goodwin, Marjorie & Charles Goodwin. 1986. Gesture and co-participation in the activity of searching for a word. Semiotica 62(1-2). 51–75 in Google Scholar

Goodwin, Charles. 2000. Action and embodiment within situated human interaction. Journal of Pragmatics 32(10). 1489–1522 in Google Scholar

Goodwin, Charles. 2003. Pointing as situated practice. In Sotaro Kita (ed.), Pointing: Where language, culture and cognition meet, 217–241. Mahwah, NJ: Erlbaum.Search in Google Scholar

Goodwin, Charles. 2007. Environmentally coupled gestures. In Susan D. Duncan, Justine Cassel & Elena T. Levy (eds.), Gesture and the dynamic dimension of language, 195–212. Amsterdam: John Benjamins Publishing Company.10.1075/gs.1.18gooSearch in Google Scholar

Green, E. Mara. 2015. One language, or maybe two: Direct communication, understanding, and informal interpreting in international deaf encounters. In. Michele Friedner & Annelies Kusters (eds.), It’s a small world: International deaf spaces and encounters, 70–82. Washington, DC: Gallaudet University Press.10.2307/j.ctv2rcnnjs.10Search in Google Scholar

Groeber, Simone & Evelyne Pochon-Berger. 2014. Turns and turn-taking in sign language interaction: A study of turn-final holds. Journal of Pragmatics 65. 121–136. in Google Scholar

Heritage, John & Steven Clayman. 2010. Talk in action: Interactions, identities, and institutions. Chichester: Wiley-Blackwell.10.1002/9781444318135Search in Google Scholar

Hiddinga, Anja & Onno Crasborn. 2011. Signed languages and globalization. Language in Society 40(4). 483–505 in Google Scholar

Hutchins, Edwin & Saeko Nomura. 2011. Collaborative construction of multimodal utterances. In Jürgen Streeck, Charles Goodwin & Curtis, LeBaron (eds.), Embodied interaction: Language and body in the material world, 29–42. Cambridge: Cambridge University Press.Search in Google Scholar

Jefferson, Gail. 2004. Glossary of transcript symbols with an introduction. In Gene H. Lerner (ed.), Conversation analysis. Studies from the first generation, 13–31. Amsterdam: John Benjamins.10.1075/pbns.125.02jefSearch in Google Scholar

Jewitt, Carey. 2004. Multimodality and new communication technologies. In Philip LeVine & Ron Scollon (eds.), Discourse & technology: Multimodal discourse analysis, 184–195. Washington, D.C.: Georgetown University Press.Search in Google Scholar

Jewitt, Carey. 2008. Multimodality and literacy in school classrooms. Review of Research in Education 32(1). 241–267. in Google Scholar

Johnston, Trevor. 2016. Auslan corpus annotation guidelines. Melbourne: Macquarie University (Sydney) & La Trobe University. Available at: (accessed 25 October 2019).Search in Google Scholar

Kendon, Adam. 2014. Semiotic diversity in utterance production and the concept of ‘language’. Philosophical Transactions of the Royal Society of London 369. 20130293. 1–13. in Google Scholar

Kramsch, Claire (ed.). 2002. Language acquisition and language socialization: Ecological perspectives. London: Continuum.Search in Google Scholar

Kress, Gunther & Theo van Leeuwen. 2001. Multimodal discourse: The modes and media of contemporary communication. Oxford: Oxford University Press.Search in Google Scholar

Kusters, Annelies. 2017. Gesture-based customer interactions: Deaf and hearing Mumbaikars’ multimodal and metrolingual practices. International Journal of Multilingualism 14(3). 283–302. in Google Scholar

Laakso, Minna. 2011. Kun sanat eivät riitä kertomaan: Eleet afasiakeskusteluissa ja lasten vuorovaikutustilanteissa [when words are not enough to tell: Gestures in aphasia conversation and children’s interaction]. In Pentti Haddington & Leila Kääntä (eds.), Kieli, keho ja vuorovaikutus. Multimodaalinen näkökulma sosiaaliseen toimintaan, 152–173. Helsinki: Suomalaisen Kirjallisuuden Seura.Search in Google Scholar

Leskelä, Leealaura & Camilla Lindholm. 2012. Haavoittuva keskustelu: Keskustelunanalyyttisia tutkimuksia kielellisesti epäsymmetrisestä vuorovaikutuksesta [Vulnerable conversation: Conversation analytic studies of linguistically asymmetric interaction]. Helsinki: Kehitysvammaliitto.Search in Google Scholar

Liddell, Scott K. 2003. Grammar, gesture, and meaning in American sign language. Cambridge: Cambridge University Press.10.1017/CBO9780511615054Search in Google Scholar

McKee, Rachel Locker & Jemina Napier. 2002. Interpreting into international sign pidgin: An analysis. Sign Language and Linguistics 5(1). 27–54. in Google Scholar

McNeill, David. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Search in Google Scholar

Messina Dahlberg, Giulia & Sangeeta Bagga-Gupta. 2019. On the quest to “go beyond” a bounded view of language. Research in the intersections of the educational sciences, language studies and deaf studies domains 1997–2018. Deafness and Educational International 21(2-3). 74–98. in Google Scholar

Mondada, Lorenza. 2014. The local constitution of multimodal resources in social interaction. Journal of Pragmatics 65. 137–156. in Google Scholar

Mondada, Lorenza. 2019. Contemporary issues in conversation analysis: Embodiment and materiality, multimodality and multisensoriality in social interaction. Journal of Pragmatics 145. 47–62. in Google Scholar

Määttä, Simo K. 2015. Interpreting the discourse of reporting: The case of screening interviews with asylum seekers and police interviews in Finland. Translation & Interpreting 7(3). 21–35.Search in Google Scholar

Norris, Sigrid. 2004. Analyzing multimodal interaction: A methodological framework. London: Routledge.10.4324/9780203379493Search in Google Scholar

Ortega, Lourdes. 2019. SLA and the study of equitable multilingualism. The Modern Language Journal 103. 23–38. in Google Scholar

Paananen, Jenny. 2015. Kuinka eleet helpottavat yhteisymmärrystä? Ikoniset ja deiktiset eleet monikulttuurisilla lääkärin vastaanotolla. [How does gesturing facilitate mutual understanding? Iconic and deictic gestures in multicultural general practice consultations]. Puhe ja Kieli 35(2). 73–95.Search in Google Scholar

Puumala, Eeva, Riitta Ylikomi & Hanna-Leena Ristimäki. 2017. Giving an account of persecution: The dynamic formation of asylum narratives. Journal of Refugee Studies, 31(2). 197–215. in Google Scholar

Puupponen, Anna, Tuija Wainio, Birgitta Burger & Tommi Jantunen. 2015. Head movements in Finnish Sign Language on the basis of motion capture data: A study of the form and function of nods, nodding, head thrusts, and head pulls. Sign Language and Linguistics 18. 41–89. in Google Scholar

Rosenstock, Rachel & Jemina Napier. 2016. International sign: Linguistic, usage, and status issues. Washington DC: Gallaudet University Press.10.2307/j.ctv2t5xgp9Search in Google Scholar

Sacks, Harvey, Emanuel A. Schegloff & Gail Jefferson. 1974. A simplest systematic for the organization of turn-taking for conversation. Language 50(4). 696–735. in Google Scholar

Schegloff, Emanuel A., Gail Jefferson & Harvey, Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Language 53(2). 361–382. in Google Scholar

Shaw, Sara, Fiona Copland & Julia Snell. 2015. An introduction to linguistic ethnography: Interdisciplinary explorations. In Julia Snell, Sara Shaw & Fiona Copland (eds.), Linguistic ethnography: Interdisciplinary explorations, 1–13. Basingstoke: Palgrave Macmillan.10.1057/9781137035035_1Search in Google Scholar

Sivunen, Nina. 2019. An ethnographic study of deaf refugees seeking asylum in Finland. Societies 9(1). 2.10.3390/soc9010002Search in Google Scholar

Stevanovic, Melisa. 2016. Sosiaaliset rakenteet [Social structures]. In Melisa Stevanovic & Camilla Lindholm (eds.), Keskustelunanalyysi: Kuinka tutkia sosiaalista toimintaa ja vuorovaikutusta [Conversation analysis: How to study social action and interaction], 200–221. Tampere: Vastapaino.Search in Google Scholar

Streeck, Jürgen, Charles Goodwin & Curtis LeBaron. 2011. Embodied interaction in the material world: An introduction. In Jürgen Streeck, Charles Goodwin & Curtis, LeBaron (eds.), Embodied interaction: Language and body in the material world, 1–26. Cambridge: Cambridge University Press.Search in Google Scholar

Tapio, Elina. 2013. A nexus analysis of English in the everyday life of FinSL signers: A multimodal view on interaction. Oulu: University of Oulu dissertation.Search in Google Scholar

Tapio, Elina. 2014. The marginalisation of finely tuned semiotic practices and misunderstandings in relation to (signed) languages and deafness. Multimodal Communication 3(2). 131.10.1515/mc-2014-0010Search in Google Scholar

Tapio, Elina. 2019. Developing picture communication for interactional situations at the beginning of the asylum process; mapping interactional practices. Apples Journal of Applied Language Studies 13(3). 131–138.10.17011/apples/urn.201910224569Search in Google Scholar

Whynot, Lori. 2016. Understanding international sign. A sociolinguistic study. Washington, DC: Gallaudet University Press.10.2307/j.ctv2rr3fntSearch in Google Scholar

Published Online: 2020-09-21
Published in Print: 2022-11-25

© 2020 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 30.11.2023 from
Scroll to top button