Peirce’s universal categories: On their potential for gesture theory and multimodal analysis

Irene Mittelberg 1
  • 1 Dept. of English, American and Romance Studies, RWTH Aachen University, 52062, Aachen, Germany
Irene Mittelberg
  • Corresponding author
  • Dept. of English, American and Romance Studies, RWTH Aachen University, Aachen, 52062, Germany
  • Email
  • Search for other articles:
  • degruyter.comGoogle Scholar

Abstract

This paper presents an account of how Peirce’s Universal Categories (UCs) of perception and experience may, as heuristic principles, inform gesture theory and multimodal analysis. Peirce’s UCs – Firstness (possibility), Secondness (actuality), and Thirdness (law, habit) – constitute the core of his phenomenology and thus also the foundation of his triadic semiotics. I argue that compared to the basic sign-object relations icon, index, symbol mainly used in previous gesture research, the more fundamental UCs allow one to discern additional facets of how coverbal gestures act as signs. This notably pertains to the phenomenology, multidimensionality, and multifunctionality of gesture. The guiding assumption is that compared to Thirdness-laden linguistic symbols constituting written, spoken or signed discourses, gestures may exhibit the UCs to more strongly varying degrees and in different, modality-specific ways. The multimodal analyses discussed in the paper show how Firstness tends to draw attention to the articulatory qualities of gestural signs, including aesthetic and affective strata, Secondness to their experiential grounding and contextualized meaning, and Thirdness to embodied habits of perceiving, feeling, (inter-)acting, thinking, and communicating with others. I further suggest that particularly through interacting with embodied image schemata and force dynamics, such habits may give rise to flexible regularities and schematicity in gesture.

  • Andrén, Mats. 2010. Children’s gestures from 18 to 30 months. Lund: Centre for Languages and Literatures, Lund University.

  • Bakhtin, Mikhail M. 1981. The dialogic imagination: Four essays, Michael Holquist (ed.). Austin & London: University of Texas Press.

  • Bavelas, Janet, Nicole Chovil, Jan L. Coates & Lori Roe. 1995. Gestures specialized for dialogue. Personality and Social Psychology Bulletin 21. 394–405.

    • Crossref
    • Export Citation
  • Bourdieu, Pierre. 1980. Le sens pratique [The logic of practice]. Paris: Éditions de Minuit.

  • Bressem, Jana. 2013. A linguistic perspective on the notation of form features in gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Tessendorf (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 1079–1098. Berlin: Mouton de Gruyter.

  • Bressem, Jana. 2014. Repetitions in gesture. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 1037–1958. Berlin: Mouton de Gruyter.

  • Bressem, Jana & Cornelia Müller. 2014. A repertoire of German recurrent gestures with pragmatic functions. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 1575–1591. Berlin: Mouton de Gruyter.

  • Calbris, Geneviève. 2011. Elements of meaning in gesture. Amsterdam & Philadelphia: John Benjamins.

  • Cienki, Alan. 2013a. Cognitive linguistics: Spoken language and gesture as expressions of conceptualization. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Tessendorf (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 182–201. Berlin: Mouton de Gruyter.

  • Cienki, Alan. 2013b. Mimetic schemas and image schemas in cognitive linguistics and gesture studies. Review of Cognitive Linguistics 11(2). 417–432.

    • Crossref
    • Export Citation
  • Cienki, Alan & Cornelia Müller (eds.). 2008. Metaphor and gesture. Amsterdam & Philadelphia: John Benjamins.

  • Clark, Herbert H. 1996. Using language. Cambridge: Cambridge University Press.

  • Colapietro, Vincent. 2001. A lantern for the feet of inquirers: The heuristic function of the Peircean categories. Semiotica 136(1/4). 201–216.

  • Colapietro, Vincent. 2008. Peirce’s categories and sign studies. In Susan Petrilli (ed.), Approaches to communication: Trends in global communication studies, 43–58. Madison, WI: Atwood.

  • Corrington, Robert. 1993. An introduction to C. S. Peirce: Philosopher, semiotician, and ecstatic naturalist. Lanham, MD: Rowman & Littlefield.

  • Danaher, David. 1998. Peirce’s semiotic and cognitive metaphor theory. Semiotica 119(1/2). 171–207.

  • Dancygier, Barbara & Eve Sweetser. 2014. Figurative language. Cambridge: Cambridge University Press.

  • Deacon, Terrence. 1997. The symbolic species: The co-evolution of language and the brain. New York & London: Norton.

  • Enfield, Nicolas. 2009. The anatomy of meaning. Speech, gestures, and composite utterances. Cambridge: Cambridge University Press.

  • Enfield, Nicolas. 2011. Elements of formulation. In Jürgen Streeck, Charles Goodwin & Curtis LeBaron (eds.), Embodied interaction: Language and body in the material world, 59–66. Cambridge: Cambridge University Press.

  • Farias, Priscila & João Queiroz. 2006. Images, diagrams, and metaphors: Hypoicons in the context of Peirce’s sixty-six-fold classification of signs. Semiotica 162(1/4). 287–307.

  • Farias, Priscila & João Queiroz. 2017. Visualizing triadic relations: Diagrams for Charles S. Peirce’s classifications of signs. Information Design Journal 23(2). 127–147.

    • Crossref
    • Export Citation
  • Fillmore, Charles J. 1982. Frame semantics. In Linguistic Society of Korea (ed.), Linguistics in the morning calm, 111–137. Seoul: Hanshin.

  • Fricke, Ellen. 2007. Origo, Geste und Raum – Lokaldeixis im Deutschen [Origo, gesture and space – spatial deixis in German]. Berlin & New York: Mouton de Gruyter.

  • Fricke, Ellen. 2012. Grammatik multimodal: Wie Wörter und Gesten zusammenwirken [Grammar multimodal: How words and gestures intertwine]. Berlin: Mouton de Gruyter.

  • Gallagher, Shaun. 2005. How the body shapes the mind. Oxford: Oxford University Press.

  • Gibbs, Raymond W. 2006. Embodiment and cognitive science. Cambridge: Cambridge University Press.

  • Goldberg, Adele E. 1995. Constructions: A construction grammar approach to argument structure. Chicago: University of Chicago Press.

  • Goodwin, Charles. 2011. Contextures of action. In Jürgen Streeck, Charles Goodwin & Curtis LeBaron (eds.), Embodied interaction: Language and the body in the material world, 182–193. Cambridge: Cambridge University Press.

  • Halton, Eugene. 2004. The living gesture and the signifying moment. Symbolic Interaction 27(1). 89–113.

    • Crossref
    • Export Citation
  • Haviland, John. 2000. Pointing, gesture spaces, and mental maps. In David McNeill (ed.), Language and gesture, 13–46. Cambridge: Cambridge University Press.

  • Jäger, Ludwig & Erika Linz (eds.). 2004. Medialität und Mentalität: Theoretische und empirische Studien zum Verhältnis von Sprache, Subjektivität und Kognition [Mediality and mind: Theoretical and empirical studies on the relation of language, subjectivity, and cognition]. München: Wilhelm Fink Verlag.

  • Jakobson, Roman. 1987 [1960]. Linguistics and poetics. In Krystyna Pomorska & Stephen Rudy (eds.), Roman Jakobson, language in literature, 62–94. Cambridge & London: Belknap Press of Harvard University Press.

  • Jakobson, Roman. 1990 [1966]. ​Quest for the essence of language. In Linda R. Waugh & Monique Monville-Burston (eds.), Roman Jakobson: On language, 407–421. Cambridge & London: Belknap Press of Harvard University Press.

  • Johnson, Mark. 1987. The body in the mind: The bodily basis of reason and imagination. Chicago: Chicago University Press.

  • Johnson, Mark. 2007. The meaning of the body: Aesthetics of human understanding. Chicago: Chicago University Press.

  • Kappelhoff, Hermann & Cornelia Müller. 2011. Embodied meaning construction. Multimodal metaphor and expressive movement in speech, gesture, and in feature film. Metaphor and the Social Word 1(2). 121–153.

    • Crossref
    • Export Citation
  • Kendon, Adam. 2004. Gesture: Visible action as utterance. Cambridge: Cambridge University Press.

  • Kita, Sotaro (ed.). 2003. Pointing: Where language, culture and cognition meet. Mahwah, NJ: Erlbaum.

  • Klee, Paul. 1986. Pedagogical sketchbook. London: Faber and Faber.

  • Koch, Sabine C., Stefanie Glawe & Daniel V. Holt. 2011. Up and down, front and back: Movement and meaning in the vertical and sagittal axes. Social Psychology 42(3). 215–224.

  • Kockelman, Paul. 2005. The semiotic stance. Semiotica 157(1/4). 233–304.

  • Krois, John M., Mats Rosengren, Angela Steidele & Dirk Westerkamp (eds.). 2007. Embodiment in cognition and culture. Amsterdam & Philadelphia: John Benjamins.

  • Ladewig, Silva H. 2014. Recurrent gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 1558–1574. Berlin: Mouton de Gruyter.

  • Lakoff, George & Mark Johnson. 1980. Metaphors we live by. Chicago: Chicago University Press.

  • Lakoff, George & Mark Johnson. 1999. Philosophy in the flesh: The embodied mind and its challenge to Western thought. New York: Basic Books.

  • Levin, Kasper. 2016. Aesthetic movements of embodied minds: Between Merleau-Ponty and Deleuze. Continental Philosophy Review 49(2). 181–202.

    • Crossref
    • Export Citation
  • Liddell, Scott. 2003. Grammar, gesture and meaning in American Sign Language. Cambridge: Cambridge University Press.

  • Liszka, James J. 1996. A general introduction to the semiotic of Charles Sanders Peirce. Bloomington, IN: Indiana University Press.

  • McNeill, David. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.

  • McNeill, David. 2005. Gesture and thought. Chicago: University of Chicago Press.

  • Merleau-Ponty, Maurice. 1962. Phenomenology of perception. New York: Humanities Press.

  • Mittelberg, Irene. 2006. Metaphor and metonymy in language and gesture: Discourse evidence for multimodal models of grammar. Ann Arbor, MI: Cornell University dissertation.

  • Mittelberg, Irene. 2008. Peircean semiotics meets conceptual metaphor: Iconic modes in gestural representations of grammar. In Alan Cienki & Cornelia Müller (eds.), Metaphor and gesture, 115–154. Amsterdam & Philadelphia: John Benjamins.

  • Mittelberg, Irene. 2010a. Interne und externe Metonymie: Jakobsonsche Kontiguitätsbeziehungen in redebegleitenden Gesten [Internal and external metonymy: Jakobsonian contiguity relations in co-speech gestures]. Sprache und Literatur 41(1). 112–143.

    • Crossref
    • Export Citation
  • Mittelberg, Irene. 2010b. Geometric and image-schematic patterns in gesture space. In Vyvyan Evans & Paul Chilton (eds.), Language, cognition, and space: The state of the art and new directions, 351–385. London: Equinox.

  • Mittelberg, Irene. 2011. Focus on form: Reflections on the (neuro-) aesthetics of abstraction in painting and gesture. In Karin Herrmann (ed.), Neuroästhetik: Perspektiven auf ein interdisziplinäres Forschungsgebiet, 110–120. Kassel: Kassel University Press.

  • Mittelberg, Irene. 2013a. The exbodied mind: Cognitive-semiotic principles as motivating forces in gesture. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Tessendorf (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 750–779. Berlin: Mouton de Gruyter.

  • Mittelberg, Irene. 2013b. Balancing acts: Image schemas and force dynamics as experiential essence in pictures by Paul Klee and their gestural enactments. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the creative mind, 325–346. Stanford: CSLI.

  • Mittelberg, Irene. 2017a. Experiencing and construing spatial artifacts from within: Simulated artifact immersion as a multimodal viewpoint strategy. Cognitive Linguistics 28(3). 381–415.

    • Crossref
    • Export Citation
  • Mittelberg, Irene. 2017b. Embodied frames and scenes: Body-based metonymy and pragmatic inferencing in gesture. Gesture 16(2). 203–244.

    • Crossref
    • Export Citation
  • Mittelberg, Irene. 2018. Gestures as image schemas and force gestalts: A dynamic-systems approach augmented with motion-capture data analyses. Cognitive Semiotics 11(1). DOI: 10.1515/cogsem-2018-0002

  • Mittelberg, Irene. 2019. Visuo-kinetic signs are inherently metonymic: How embodied metonymy motivates forms, functions, and schematic patterns in gesture. Research topic “Visual Language,” Frontiers in Psychology.

  • Mittelberg, Irene, Thomas H. Schmitz & Hannah Groninger. 2017. Operative manufacts: Gestures as embodied sketches in the early stages of the design process. In Sabine Ammon & Remei Capdevila-Werning (eds.), The active image: Architecture and engineering in the age of modeling, 99–131. Berlin & New York: Springer.

  • Mittelberg, Irene & Daniel Schüller. 2016. Kulturwissenschaftliche Orientierung in der Gestenforschung [Taking a cultural studies orientation in gesture research]. In Ludwig Jäger, Werner Holly, Peter Krapp & Samuel Weber (eds.), Language – Culture – Communication: An international handbook of linguistics as cultural study, 871–884. Berlin & New York: Mouton de Gruyter.

  • Mittelberg, Irene & Linda R. Waugh. 2009. Metonymy first, metaphor second: A cognitive-semiotic approach to multimodal figures of thought in co-speech gesture. In Charles Forceville & Eduardo Urios-Aparisi (eds.), Multimodal metaphor, 329–356. Berlin & New York: Mouton de Gruyter.

  • Mittelberg, Irene & Linda R. Waugh. 2014. Gestures and metonymy. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 1747–1766. Berlin: Mouton de Gruyter.

  • Müller, Cornelia. 1998. Redebegleitende Gesten. Kulturgeschichte – Theorie – Sprachvergleich [Co-speech gestures: Cultural history – theory – cross-linguistic insights]. Berlin: Berlin Verlag.

  • Müller, Cornelia. 2010. Wie Gesten bedeuten. Eine kognitiv-linguistische und sequenzanalytische Perspektive [How gestures mean: A cognitive linguistic and sequence-analytical perspective]. Sprache und Literatur 41(1). 37–68.

    • Crossref
    • Export Citation
  • Müller, Cornelia. 2017. How recurrent gestures mean: Conventionalized contexts of use and embodied motivation. Gesture 16(2). 277–304.

    • Crossref
    • Export Citation
  • Müller, Cornelia, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.). 2014. Body-language-communication: An international handbook on multimodality in human interaction, Vol. 2. Berlin & Boston: Mouton De Gruyter.

  • Müller, Cornelia, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Tessendorf (eds.). 2013. Body-language-communication: An international handbook on multimodality in human interaction, Vol. 1. Berlin: Mouton de Gruyter.

  • Nöth, Winfried. 2016. Habits, habit of change, and the habit of habit change according to Peirce. In Donna West & Myrdene Anderson (eds.), Consensus on Peirce’s concept of habit, 35–63. Dordrecht: Springer.

  • Núñez, Rafael & Eve Sweetser. 2006. With the future behind them: Convergent evidence from Aymara language and gesture in the crosslinguistic comparison of spatial construals of time. Cognitive Science 30. 401–450.

    • Crossref
    • PubMed
    • Export Citation
  • Oehler, Klaus. 1987. An outline of Peirce’s semiotics. In Martin Krampen, Klaus Oehler, Roland Posner, Thomas A. Sebeok & Thure von Uexküll (eds.), Classics of semiotics, 1–21. New York: Plenum Press.

    • Crossref
    • Export Citation
  • Pape, Helmut. 1990. Charles S. Peirce on objects of thought and representation. Noûs 24(3). 375–395.

    • Crossref
    • Export Citation
  • Pape, Helmut. 2015. Charles S. Peirce zur Einführung, 2nd. Hamburg: Junius.

  • Parrill, Fey. 2009. Dual viewpoint gestures. Gesture 9(3). 271–289.

    • Crossref
    • Export Citation
  • Peirce, Charles & Lady Victoria Welby. 1977. Semiotic and significs, Charles S. Hardwick (ed.). Bloomington: Indianapolis University Press. [Reference to this work will be designated SS followed by page number.].

  • Peirce, Charles S. 1931–1966. The collected papers of Charles S. Peirce, Vol. 8, C. Hartshorne, P. Weiss & A. W. Burks (eds.). Cambridge: Harvard University Press. [Reference to Peirce’s papers will be designated CP followed by volume and paragraph number.].

  • Peirce, Charles S. 1992. Essential Peirce: Selected philosophical writings, Vol. 1 (1867–1893), N. Houser & C. Kloesel (eds.). Bloomington: Indiana University Press. [Reference to vol. 1 of Essential Peirce will be designated EP 1.].

  • Peirce, Charles S. 1998. Essential Peirce: Selected philosophical writings, Vol. 2 (1893–1913), Peirce Edition Project (eds.). Bloomington: Indiana University Press. [Reference to vol. 2 of Essential Peirce will be designated EP 2.].

  • Peirce, Charles Sanders. 1955. Philosophical writings of Peirce, Justus Buchler (ed.). New York: Dover.

  • Potter, Vincent. 1967. Charles S. Peirce. On norms & ideals. Worcester, MA: University of Massachusetts Press.

  • Rekittke, Linn-Marlen, Irene Mittelberg & Dhana Wolf. 2015. How sound moves the body: Felt qualities of experience in gestural enactments of film sound. In Proceedings of the 3rd Gesture and Speech in Interaction (GESPIN) Conference, Nantes.

  • Saussure, Ferdinand de. 1986. Cours de linguistique générale de Ferndinand de Saussure, Charles Bally, Albert Sechehaye & Tuillo De Mauro (eds.). Paris: Payot.

  • Schüller, Daniel & Irene Mittelberg. 2016. Diagramme von Gesten. Eine zeichentheoretische Analyse digitaler Bewegungsspuren [Diagrams of gestures: A semiotic analysis of digital motion traces]. Zeitschrift Für Semiotik 38(3–4). 3–32.

  • Shapiro, Michael. 1983. The sense of grammar: Language as semeiotic. Bloomington, IN: Indiana University Press.

  • Sonesson, Göran. 2007. The extensions of man revisited: From primary to tertiary embodiment. In John M. Krois, Mats Rosengren, Angela Steidele & Dirk Westerkamp (eds.), Embodiment in cognition and culture, 27–53. Amsterdam: John Benjamins.

  • Sonesson, Göran. 2013. The natural history of branching: Approaches to the phenomenology of firstness, secondness, and thirdness. Signs and Society 1(2). 297–325.

    • Crossref
    • Export Citation
  • Sonesson, Göran. 2016. Thirdness as the observer observed: From habit to law by way of habitus. In Donna West & Myrdene Anderson (eds.), Consensus on Peirce’s concept of habit, 283–296. Dordrecht: Springer.

  • Stjernfelt, Frederik. 2007. Diagrammatology: An investigation on the borderlines of phenomenology, ontology, and semiotics. Dordrecht: Springer.

  • Stjernfelt, Frederik. 2014. Natural propositions: The actuality of Peirce’s doctrine of dicisigns. Boston: Docent Press.

  • Stjernfelt, Frederik. 2016. Dicisigns and habits: Implicit propositions and habit-taking in Peirce’s pragmatism. In Donna West & Myrdene Anderson (eds.), Consensus on Peirce’s concept of habit, 241–262. Dordrecht: Springer.

  • Streeck, Jürgen. 2009. Gesturecraft: The manu-facture of meaning. Amsterdam & Philadelphia: John Benjamins.

  • Streeck, Jürgen, Charles Goodwin & Curtis LeBaron (eds.). 2011. Embodied interaction: Language and the body in the material world. Cambridge: Cambridge University Press.

  • Sweetser, Eve. 2007. Looking at space to study mental spaces: Co-speech gesture as a crucial data source in cognitive linguistics. In Monica Gonzalez-Marquez, Irene Mittelberg, Seana Coulson & Michael Spivey (eds.), Methods in cognitive linguistics, 201–224. Amsterdam & Philadelphia: John Benjamins.

  • Sweetser, Eve. 2012. Introduction: Viewpoint and perspective in language and gesture, from the ground down. In Barbara Dancygier & Eve E. Sweetser (eds.), Viewpoint in language: A multimodal perspective, 1–22. Cambridge: Cambridge University Press.

    • Crossref
    • Export Citation
  • Talmy, Leonard. 1988. Force dynamics in language and cognition. Cognitive Science 12(1). 49–100.

    • Crossref
    • Export Citation
  • Vygotsky, Lev. 1962. Thought and language. Cambridge, MA: MIT Press.

  • Waugh, Linda R. 1976. Roman Jakobson’s science of language. Lisse: Peter de Ridder Press.

  • Waugh, Linda R. 1985. The poetic function and the nature of language. Poetics Today 2. 57–82.

  • West, Donna E. & Myrdene Anderson (eds.). 2016. Consensus on Peirce’s concept of habit: Before and beyond consciousness. Dordrecht: Springer.

  • Wolf, Dhana, Irene Mittelberg, Linn-Marlen Rekittke, Saurabh Bhavsar, Mikhail Zvyagintsev, Annina Haeck, Fengyu Cong, Martin Klasen & Mathiak Klaus. 2018. Interpretation of social interactions: Functional imaging of cognitive-semiotic categories during naturalistic viewing. Frontiers in Human Neuroscience 12. DOI: 10.3389/fnhum.2018.00296

    • PubMed
    • Export Citation
  • Wolf, Dhana, Linn-Marlen Rekittke, Irene Mittelberg, Martin Klasen & Klaus Mathiak. 2017. Perceived conventionality in co-speech gestures involves the fronto-temporal language network. Frontiers of Human Neuroscience. DOI: 10.3389/fnhum.2017.00573

  • Zima, Elisabeth & Alexander Bergs (eds.). 2017. Toward a multimodal construction grammar. Special issue. Linguistics Vanguard 3(s1).

  • Zlatev, Jordan. 2005. What’s in a schema? Bodily mimesis and the grounding of language. In Beate Hampe (ed.), From perception to meaning: Image schemas in cognitive linguistics, 313–342. Berlin/New York: Mouton de Gruyter.

  • Zlatev, Jordan. 2014. Image schemas, mimetic schemas, and children’s gestures. Cognitive Semiotics 7(1). 3–29.

    • Crossref
    • Export Citation
Purchase article
Get instant unlimited access to the article.
$42.00
Log in
Already have access? Please log in.


or
Log in with your institution

Journal + Issues

Search