Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter Mouton November 18, 2020

RoboDoc: Semiotic resources for achieving face-to-screenface formation with a telepresence robot

  • Brian L. Due EMAIL logo
From the journal Semiotica


Face-to-face interaction is a primordial site for human activity and intersubjectivity. Empirical studies have shown how people reflexively exhibit a face orientation and work to establish a formation in which everyone is facing each other in local participation frameworks. The Face has also been described by, e.g., Levinas as the basis for a first ethical philosophy. Humans have established these Face-formations when interacting since time immemorial, but what happens when one of the participants is present through a telepresence robot? Based on ethnomethodology, Peircean/Goodwinian semiotics, multimodal conversation analysis and video data from a Danish residential rehabilitation center, the article shows the ways in which participants manage to interactively, cooperatively, and moment by moment achieve an F-formation in situ. The article contributes a detailed analysis and discussion of the kind of participant a telepresence robot is, in and through situated interactions: I propose that we term this participant the RoboDoc, given that it is an assemblage of a doctor who controls a robot. By focusing on the affordances of mobility, the article contributes to a renewed understanding of the importance and relevance of establishing Face-orientations in an increasingly technofied telepresence world.

Corresponding author: Brian L. Due, University of Copenhagen, Copenhagen, Denmark, E-mail:


Data for this paper is from the research project “Professionals’ use of video meeting” supported by the Velux Foundation, Denmark. Special thanks to Rikke Nielsen for helping with data collection. Earlier versions of the paper were presented at the conferences Nordisco 2018 and Lansi 2018.


Arminen, Ilkka, Christian Licoppe & Anna Spagnolli. 2016. Respecifying mediated interaction. Research on Language and Social Interaction 49(4). 290–309. in Google Scholar

Arundale, Robert B. 2013. Face as a research focus in interpersonal pragmatics: Relational and emic perspectives. Journal of Pragmatics 58. 108–120. in Google Scholar

Bateson, Gregory & Margaret Mead. 1942. Balinese character, a photographic analysis. New York: New York Academy of Sciences.Search in Google Scholar

Birdwhistell, Ray. 1970. Kinesics and context, essays on body motion communication. Philadelphia: University of Pennsylvania Press.10.9783/9780812201284Search in Google Scholar

Broth, Mathias & Leelo Keevallik. 2014. Getting ready to move as a couple accomplishing mobile formations in a dance class. Space and Culture 17(2). 107–121. in Google Scholar

Callon, Michel & John Law. 1997. After the individual in society: Lessons on collectivity from science, technology, and society. Canadian Journal of Sociology/Cahiers Canadiens de Sociologie 22(2). 165–182. in Google Scholar

Clark, Andy. 2004. Natural-born cyborgs: Minds, technologies, and the future of human intelligence. Oxford: Oxford University Press.Search in Google Scholar

Clark, Andy & David Chalmers. 1998. The extended mind. Analysis 58(1). 7–19. in Google Scholar

Clynes, Manfred E. & Nathan S. Kline. 1960. Cyborgs and space. Astronautics 5(9). 26–27, 74–76.Search in Google Scholar

Dant, Tim. 2004. The driver-car. Theory, Culture & Society 21(4–5). 61–79. in Google Scholar

De Stefani, Elwys, Mathias Broth & Arnulf Deppermann. 2019. On the road: Communicating traffic. Language & Communication 65. 1–6. in Google Scholar

Deppermann, Arnulf. 2018. Chuck Goodwin’s two ground-breaking contributions to the study of social interaction: Simultaneities in multimodal interaction and professional vision. In Donald Favareau (ed.), Co-operative engagements of intertwined semiosis: Essays in honour of Charles Goodwin, 55–69. Tartu: University of Tartu Press.Search in Google Scholar

Dreyfus, Hubert. 2000. Telepistemology: Descartes’ last stand. In Ken Goldberg (ed.), The robot in the garden: Telerobotics and telepistemology in the age of the Internet, 48–63. Cambridge: MIT Press.Search in Google Scholar

Due, Brain L. 2014. The development of an idea in a context of rejection. Semiotica 202(1–4). 207–239.10.1515/sem-2014-0036Search in Google Scholar

Due, Brain L. 2017. Multimodal interaktionsanalyse og videoetnografisk dataindsamling. Region Hovedstaden: Samfundslitteratur.Search in Google Scholar

Due, Brain L. & Simon Lange. 2018a. Semiotic resources for navigation: A video ethnographic study of blind people’s uses of the white cane and a guide dog for navigating in urban areas. Semiotica 222(1–4). 287–312. in Google Scholar

Due, Brain L. & Simon Lange. 2018b. The Moses effect: The spatial hierarchy and joint accomplishment of a blind person navigating. Space and Culture 21(2). 129–144. in Google Scholar

Due, Brain L. & Johan Trærup. 2018. Passing glasses: Accomplishing deontic stance at the optician. Social Interaction 1(2). in Google Scholar

Enfield, Nick J. 2006. Social consequences of common ground. In N. J. Enfield & S. C. Levinson (eds.), Roots of human sociality: Culture, cognition, and interaction, 223–254. Oxford: Berg Press.10.4324/9781003135517-20Search in Google Scholar

Enfield, Nick J. 2009. The anatomy of meaning: Speech, gesture, and composite utterances. Cambridge: Cambridge University Press.10.1017/CBO9780511576737Search in Google Scholar

Enfield, Nick J. 2011. Elements of formulation. In J. Streeck, C. Goodwin & C. LeBaron (eds.), Embodied interaction: Language and body in the material world, 59–67. Cambridge: Cambridge University Press.Search in Google Scholar

Ford, Cecilia E. & Trini Stickle. 2012. Securing recipiency in workplace meetings: Multimodal practices. Discourse Studies 14(1). 11–30. in Google Scholar

Garfinkel, Harold. 1967. Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall.Search in Google Scholar

Garfinkel, Harold. 1991. Respecification: Evidence for locally produced, naturally accountable phenomena of order, logic, reason, meaning, methods, etc. in and of the essential haecceity of immortal ordinary society (I) – an announcement of studies. In Graham Button (ed.), Ethnomethodology and the human sciences, 10–19. Cambridge: Cambridge University Press.10.1017/CBO9780511611827.003Search in Google Scholar

Garfinkel, Harold & Harvey L. Sacks. 1970. On formal structures of practical actions. In J. C. McKinney & E. A. Tiryakian (eds.), Theoretical sociology, 338–366. New York: Appleton Century Crofts.Search in Google Scholar

Gill, Virginia Teas & Felicia Roberts. 2013. Conversation analysis in medicine. In Jack Sidnell & Tanya Stivers (eds.), The handbook of conversation analysis, 575–592. Chichester: Blackwell.10.1002/9781118325001.ch28Search in Google Scholar

Goffman, Erving. 1964. The neglected situation. American Anthropologist 66(6). 133–136. in Google Scholar

Goffman, Erving. 1967. Interaction ritual: Essays on face-to-face behavior. Garden City, NY: Doubleday.Search in Google Scholar

Goffman, Erving. 1971. Relations in public: Microstudies of the public order. New York: Harper and Row.Search in Google Scholar

Goodwin, Charles. 1979. The interactive construction of a sentence in natural conversation. In G. Psathas (ed.), Everyday language: Studies in ethnomethodology, 97–121. New York: Irvington.Search in Google Scholar

Goodwin, Charles. 1981. Conversational organization: Interaction between speakers and hearers. New York: Academic Press.Search in Google Scholar

Goodwin, Charles. 1995. Co-constructing meaning in conversations with an aphasic man. Research on Language and Social Interaction 28(3). 233–260. in Google Scholar

Goodwin, Charles. 2003. The semiotic body in its environment. In J. Coupland & R. Gwyn (eds.), Discourses of the body, 19–42. New York: Palgrave Connect.10.1057/9781403918543_2Search in Google Scholar

Goodwin, Charles. 2006. Human sociality as mutual orientation in a rich interactive environment: Multimodal utterances and pointing in aphasia. In Nick Enfield and Stephen C. Levinson (eds.), Roots of human sociality, 96–125. Oxford: Berg Press.10.4324/9781003135517-5Search in Google Scholar

Goodwin, Charles. 2017. Co-operative action. Cambridge: Cambridge University Press.10.1017/9781139016735Search in Google Scholar

Goodwin, Marjorie Harness & Charles Goodwin. 2012. Car talk: Integrating texts, bodies, and changing landscapes. Semiotica 191(1/4). 257–286.10.1515/sem-2012-0063Search in Google Scholar

Greimas, Algirdas J. 1987. Actants, actors, and figures. In On meaning: Selected writings in semiotic theory (Theory and history of literature 38), Paul J. Perron & Frank H. Collins (trans.), 106–120. Minneapolis: University of Minnesota Press.Search in Google Scholar

Greimas, Algirdas J. & F. Rastier. 1968. The interaction of semiotic constraints. Yale French Studies 41. 86–105. in Google Scholar

Haddington, Pentti, Tinna Keisanen, Lorenza Mondada & Maurice Nevile. 2014. Multiactivity in social interaction: Beyond multitasking. Amsterdam: John Benjamins.10.1075/z.187Search in Google Scholar

Haddington, Pentti, Lorenza Mondada & Maurice Nevile. 2013. Interaction and mobility, language and the body in motion. Berlin: De Gruyter.10.1515/9783110291278Search in Google Scholar

Haddington, Pentti, Maurice Nevile & Tiina Keisanen. 2012. Meaning in motion: Sharing the car, sharing the drive. Semiotica 191(1–4). 101–116.10.1515/sem-2012-0057Search in Google Scholar

Hall, Edward T. 1966. The hidden dimension. New York: Anchor.Search in Google Scholar

Haraway, Donna. 1990. Simians, cyborgs, and women: The reinvention of nature. New York: Routledge.Search in Google Scholar

Haraway, Donna. 1991. A cyborg manifesto: Science, technology, and socialist-feminism in the late 20th century. In Simians, cyborgs and women: The reinvention of nature, 149–181. London: Routledge.10.1007/978-1-4020-3803-7_4Search in Google Scholar

Haraway, Donna. 2003. The companion species manifesto: Dogs, people, and significant otherness, M. Begelke (ed.). Chicago: Prickly Paradigm Press.Search in Google Scholar

Heath, Christian. 1984. Participation in the medical consultation: The co-ordination of verbal and nonverbal behaviour between the doctor and patient. Sociology of Health & Illness 6(3). 311–388. in Google Scholar

Heath, Christain, Jon Hindmarsh & Poul Luff. 2010. Video in qualitative research. London: SAGE.Search in Google Scholar

Heath, Christain & Poul Luff. 1992. Media space and communicative asymmetries: Preliminary observations of video-mediated interaction. Human–Computer Interaction 7(3). 315–346. in Google Scholar

Heidegger, Martin. 1977. The question concerning technology – and other essays. New York: Garland.Search in Google Scholar

Heritage, John & Douglas W. Maynard. 2006. Communication in medical care. Cambridge: Cambridge University Press.10.1017/CBO9780511607172Search in Google Scholar

Heritage, John & Jeffrey D. Robinson. 2006. Accounting for the visit: Giving reasons for seeking medical care. In John Heritage & Douglas W. Maynard (eds.), Communication in medical care: Interaction between primary care physicians and patients, 48–85. Cambridge: Cambridge University Press.10.1017/CBO9780511607172.005Search in Google Scholar

Hjulstad, John. 2016. Practices of organizing built space in videoconference-mediated interactions. Research on Language and Social Interaction 49(4). 325–341. in Google Scholar

Jefferson, Gail. 2004. Glossary of transcript symbols with an introduction. In Gene H. Lerner (ed.), Conversation analysis: Studies from the first generation, 13–31. Amsterdam: John Benjamins.10.1075/pbns.125.02jefSearch in Google Scholar

Kendon, Adam. 1976. The F-formation system: The spatial organization of social encounters. Man–Environment Systems 6. 291–296.Search in Google Scholar

Kockelman, Paul. 2007. Agency: The relation between meaning, power, and knowledge. Current Anthropology 48(3). 375–401. in Google Scholar

Krummheuer, Antonia Lina. 2016. Who am i? What are you? Identity construction in encounters between a teleoperated robot and people with acquired brain injury. In A. Agah, J. J. Cabibihan, A. Howard, M. Salichs & H. He (eds.), Social Robotics. ICSR 2016 (Lecture notes in computer science 9979), 880–889. Berlin: Springer.10.1007/978-3-319-47437-3_86Search in Google Scholar

Latour, Bruno. 1996a. On actor-network theory: A few clarifications. Soziale Welt 47(4). 369–381.Search in Google Scholar

Latour, Bruno. 1996b. On interobjectivity. Mind, Culture, and Activity 3(4). 228. in Google Scholar

Latour, Bruno. 2005. Reassembling the social: An introduction to actor-network-theory. Oxford: Oxford University Press.Search in Google Scholar

Law, John. 2009. Actor network theory and material semiotics. In The new Blackwell companion to social theory, 141–158. Chichester: John Wiley.10.1002/9781444304992.ch7Search in Google Scholar

Levinas, Emmanuel. 1979. Totality and infinity: An essay on exteriority. Berlin: Springer.10.1007/978-94-009-9342-6Search in Google Scholar

Lévinas, Emmanuel. 1985. Ethics and infinity, P. Nemo (trans.). Pittsburgh: Duquesne University Press.Search in Google Scholar

Liberman, Kenneth. 2013. More studies in ethnomethodology. Albany, NY: SUNY Press.Search in Google Scholar

Licoppe, Chistian. 2017. Showing objects in Skype video-mediated conversations: From showing gestures to showing sequences. Journal of Pragmatics 110. 63–82. in Google Scholar

Licoppe, Christain & Julin Figeac. 2018. Gaze patterns and the temporal organization of multiple activities in mobile smartphone uses. Human–Computer Interaction 33(5–6). 311–334. in Google Scholar

Licoppe, Christian, Paul Luff, Christian Heath, Hideaki Kuzuoka, Naomi Yamashita & Sylvane Tuncer. 2017. Showing objects: Holding and manipulating artefacts in video-mediated collaborative settings. In CHI ’17: Proceedings of the 2017 CHI conference on human factors in computing systems, 5295–5306. New York: ACM Press.10.1145/3025453.3025848Search in Google Scholar

Licoppe, Christian & Julien Morel. 2012. Video-in-interaction: ‘Talking heads’ and the multimodal organization of mobile and Skype video calls. Research on Language and Social Interaction 45. 37–41. in Google Scholar

Licoppe, Christian & Julien Morel. 2014. Mundane video directors: Showing one’s environment in Skype and mobile video calls. In M. Broth, E. Laurier & L. Mondada (eds.), Studies of video practices: Video at work, 135–160. London: Routledge.Search in Google Scholar

Licoppe, Christian & Julien Morel. 2018. Visuality, text and talk, and the systematic organization of interaction in Periscope live video streams. Discourse Studies 20(5). 637–665. in Google Scholar

Licoppe, Christian & Clair-Antonie Veyrier. 2017. How to show the interpreter on screen? The normative organization of visual ecologies in multilingual courtrooms with video links. Journal of Pragmatics 107. 147–164. in Google Scholar

Luff, Paul, Christian Heath, Hideaki Kuzuoka, Jon Hindmarsh, Keiichi Yamazaki & Shinya Oyama. 2003. Fractured ecologies: Creating environments for collaboration. Human–Computer Interaction 18(1). 51–84. in Google Scholar

Luff, Paul, Christian Heath, Naomi Yamashita, Hideaki Kuzuoka & Marine Jirotka. 2016. Embedded reference: Translocating gestures in video-mediated interaction. Research on Language and Social Interaction 49(4). 342–361. in Google Scholar

Luff, Paul, Marina Jirotka, Naomi Yamashita, Hideaki Kuzuoka, Christian Heath & Grace Eden. 2013. Embedded interaction: The accomplishment of actions in everyday and video-mediated environments. AIS Transactions on Human–Computer Interaction 20(1). 1–22. in Google Scholar

Marstrand, Ann Katrine & Jan Svennevig. 2018. A preference for non-invasive touch in caregiving contexts. Social Interaction 1(2). in Google Scholar

McGregor, Moira, Barry Brown & Donald McMillan. 2014. 100 days of iPhone use: Mobile recording in the wild. In CHI ’14 extended abstracts on human factors in computing systems, 2335–2340. New York: ACM.10.1145/2559206.2581296Search in Google Scholar

McIlvenny, Paul. 2013. Interacting outside the box: Between social interaction and mobilities. In Pentti Haddington, Lorenza Mondada & Maurice Nevile (eds.), Interaction and mobility: Language and the body in motion, 409–417. Berlin: De Gruyter.10.1515/9783110291278.409Search in Google Scholar

McIlvenny, Paul. 2018. How did the mobility scooter cross the road? Coordinating with co-movers and other movers in traffic. Language & Communication 65. 105–130.10.1016/j.langcom.2018.04.002Search in Google Scholar

McIlvenny, Paul, Mathias Broth & Pentti Haddington. 2009. Communicating place, space, and mobility. Journal of Pragmatics 41(10). 1879–1886. in Google Scholar

Merleau-Ponty, Maurice. 2002. Phenomenology of perception. London: Routledge.10.4324/9780203994610Search in Google Scholar

Meyer, Christian, Jurgen Streeck & J. Scott Jordan. 2017. Intercorporeality: Emerging socialities in interaction. Oxford: Oxford University Press.10.1093/acprof:oso/9780190210465.001.0001Search in Google Scholar

Mikesell, Lisa. 2016. The use of directives to repair embodied (mis)understandings in interactions with individuals diagnosed with frontotemporal dementia. Research on Language and Social Interaction 49(3). 201–219. in Google Scholar

Mondada, Lorenza. 2009. Emergent focused interactions in public places: A systematic analysis of the multimodal achievement of a common interactional space. Journal of Pragmatics 41(10). 1977–1997. in Google Scholar

Mondada, Lorenza. 2014. The local constitution of multimodal resources for social interaction. Journal of Pragmatics 65. 137–156. in Google Scholar

Mondada, Lorenza. 2018. Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality. Research on Language and Social Interaction 51(1). 85–106. in Google Scholar

Norman, Donald. 1993. Things that make us smart: Defending human attributes in the age of the machine. New York: Basic.Search in Google Scholar

Peirce, Charles S. 1955. Philosophical writings of Peirce, J. Buchler (ed.). New York: Dover.Search in Google Scholar

Perakyla, Anssi. 1998. Authority and accountability: The delivery of diagnosis in primary health care. Social Psychology Quarterly 61(4). 301–320. in Google Scholar

Philipsen, Johanne S. & Lasse V. Jensen. 2018. “How do signs come to mean?” – Reflections on the Goodwinian interactional approach to empirical investigations of the human semiotic ecology. In D. Favareau (ed.), Co-operative engagements of intertwined semiosis: Essays in honour of Charles Goodwin, 301–305. Tartu: University of Tartu Press.Search in Google Scholar

Ritchie, Graeme. 1999. Developing the incongruity-resolution theory. (accessed 15 September 2020).Search in Google Scholar

Robillard, Albert B. 1999. Meaning of a disability: The lived experience of paralysis. Philadelphia: Temple University Press.Search in Google Scholar

Sacks, Harvey L., Emmanuel A. Schegloff & Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. Language 50(4). 696–735. in Google Scholar

Scheflen, Albert E. 1968. Human communication: Behavioral programs and their integration in interaction. Behavioral Science 13(1). 44–55. in Google Scholar

Schegloff, Emmanuel A. 1987. Analyzing single episodes of interaction: An exercise in conversation analysis. Social Psychology Quarterly 50(2). 101–114. in Google Scholar

Schegloff, Emmanuel A. & Harvey L. Sacks. 1973. Opening up closings. Semiotica 8(4). 289–327. in Google Scholar

Sørensen, Anders Stengaard & Gitte Rasmussen. 2018. RoBody interaction: A new approach at kinesthetic human robot interaction. In Proceedings of RO-MAN 2018 – IEEE international conference on robot and human interactive communication. Piscataway, NJ: IEEE.10.1109/ROMAN.2018.8525844Search in Google Scholar

Stefani, Elvys De. 2013. The collaborative organisation of next actions in a semiotically rich environment: Shopping as a couple. In P. Haddington, L. Mondada & M. Nevile (eds.), Interaction and mobility language and the body in motion, 123–151. Berlin: De Gruyter.10.1515/9783110291278.123Search in Google Scholar

Stevanovic, Melisa & Anssi Peräkylä. 2012. Deontic authority in interaction: The right to announce, propose, and decide. Research on Language & Social Interaction 45(3). 297–321. in Google Scholar

Stivers, Tanya & Jack Sidnell. 2005. Introduction: Multimodal interaction. Semiotica 156(1/4). 1–20. in Google Scholar

Urry, John. 2007. Mobilities. Cambridge: Polity.Search in Google Scholar

Webb, Helena, Dirk vom Lehn, Christian Heath, Will Gibson & Bruce J. W. Evans. 2013. The problem with “problems”: The case of openings in optometry consultations. Research on Language and Social Interaction 46(1). 65–83. in Google Scholar

Published Online: 2020-11-18
Published in Print: 2021-01-27

© 2020 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 1.4.2023 from
Scroll to top button