Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Multimodal Communication

Ed. by Norris, Sigrid

2 Issues per year

Online
ISSN
2230-6587
See all formats and pricing
More options …

A Picture is Worth More Words Over Time: Multimodality and Narrative Structure Across Eight Decades of American Superhero Comics

Neil Cohn
  • Corresponding author
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LE Tilburg, Netherlands
  • Email:
/ Ryan Taylor
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LE Tilburg, Netherlands
/ Kaitlin Pederson
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LE Tilburg, Netherlands
Published Online: 2017-04-20 | DOI: https://doi.org/10.1515/mc-2017-0003

Abstract

The visual narratives of comics involve complex multimodal interactions between written language and the visual language of images, where one or the other may guide the meaning and/or narrative structure. We investigated this interaction in a corpus analysis across eight decades of American superhero comics (1940–2010s). No change across publication date was found for multimodal interactions that weighted meaning towards text or across both text and images, where narrative structures were present across images. However, we found an increase over time of narrative sequences with meaning weighted to the visuals, and an increase of sequences without text at all. These changes coincided with an overall reduction in the number of words per panel, a shift towards panel framing with single characters and close-ups rather than whole scenes, and an increase in shifts between temporal states between panels. These findings suggest that storytelling has shifted towards investing more information in the images, along with an increasing complexity and maturity of the visual narrative structures. This has shifted American comics from being textual stories with illustrations to being visual narratives that use text.

Keywords: visual language; multimodality; narrative structure; comics; discourse

References

  • Abbott, M., and Forceville, C. (2011). Visual representation of emotion in manga: loss of control is loss of hands in Azumanga Daioh Volume 4. Language and Literature, 20:91–112.Google Scholar

  • Bateman, J. A. (2014). Text and Image: A Critical Introduction to the Visual/Verbal Divide. New York: Routledge.Google Scholar

  • Bateman, J. A., and Wildfeuer, J. (2014a). Defining units of analysis for the systematic analysis of comics: a discourse-based approach. Studies in Comics, 5:373–403.Google Scholar

  • Bateman, J. A., and Wildfeuer, J. (2014b). A multimodal discourse theory of visual narrative. Journal of Pragmatics, 74:180–208.CrossrefGoogle Scholar

  • Brienza, C. (2016). Manga in America: transnational book publishing and the domestication of Japanese comics. London: Bloomsbury Academic.Google Scholar

  • Burgas, G. (2007). Compressed storytelling versus decompressed storytelling: pros and cons [Online]. Comic Book Resources. Available: http://www.cbr.com/compressed-storytelling-versus-decompressed-storytelling-pros-and-cons/ [Accessed 2/27/2017].

  • Cicci, M. (2015). Turning the page: Fandoms, multimodality, the transformation of the “comic book” superhero. Doctoral Dissertation, Wayne State University.Google Scholar

  • Cohn, N. (2011). A different kind of cultural frame: an analysis of panels in American comics and Japanese manga. Image [&] Narrative, 12:120–134.Google Scholar

  • Cohn, N. (2013a). The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. London, UK: Bloomsbury.Google Scholar

  • Cohn, N. (2013b). Visual narrative structure. Cognitive Science, 37:413–452.CrossrefGoogle Scholar

  • Cohn, N. (2014). The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology, 5:1–9.CrossrefGoogle Scholar

  • Cohn, N. (2015). Narrative conjunction’s junction function: the interface of narrative grammar and semantics in sequential images. Journal of Pragmatics, 88:105–132.CrossrefGoogle Scholar

  • Cohn, N. (2016). A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition, 146:304–323.CrossrefGoogle Scholar

  • Cohn, N., and Bender, P. (2017). Drawing the line between constituent structure and coherence relations in visual narratives. Journal of Experimental Psychology: Learning, Memory, & Cognition, 43:289–301.Google Scholar

  • Cohn, N., and Ehly, S. (2016). The vocabulary of manga: visual morphology in dialects of Japanese visual language. Journal of Pragmatics, 92:17–29.CrossrefGoogle Scholar

  • Cohn, N., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2014). The grammar of visual narrative: neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64:63–70.CrossrefGoogle Scholar

  • Cohn, N., and Kutas, M. (2015). Getting a cue before getting a clue: event-related potentials to inference in visual narrative comprehension. Neuropsychologia, 77:267–278.CrossrefGoogle Scholar

  • Cohn, N., and Kutas, M. (Under Review). What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing.Google Scholar

  • Cohn, N., Paczynski, M., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2012a). (Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension. Cognitive Psychology, 65:1–38.CrossrefGoogle Scholar

  • Cohn, N., Taylor-Weiner, A., and Grossman, S. (2012b). Framing attention in Japanese and American comics: cross-cultural differences in attentional structure. Frontiers in Psychology – Cultural Psychology, 3:1–12.Google Scholar

  • Coogan, P. (2006). Superhero: The Secret Origin of a Genre. Austin, TX: MonkeyBrain.Google Scholar

  • Cutting, J. E. (2015). The framing of characters in popular movies. Art & Perception, 3:191–212.CrossrefGoogle Scholar

  • Cutting, J. E., Delong, J. E., and Nothelfer, C. E. (2010). Attention and the evolution of hollywood film. Psychological Science, 21:432–439.CrossrefGoogle Scholar

  • Duncan, R., and Smith, M. J. (2009). The Power of Comics. New York: Continuum Books.Google Scholar

  • Duncan, R., Smith, M. J., and Levitz, P. (2015). The Power of Comics. New York: Continuum Books.Google Scholar

  • Forceville, C. (2011). Pictorial runes in Tintin and the Picaros. Journal of Pragmatics, 43:875–890.CrossrefGoogle Scholar

  • Forceville, C. (2016). Conceptual metaphor theory, blending theory, and other cognitivist perspectives on comics . In: The Visual Narrative Reader, N. Cohn (Ed.). Bloomsbury: London.Google Scholar

  • Forceville, C., and Urios-Aparisi, E. (2009). Multimodal Metaphor. New York: Mouton De Gruyter.Google Scholar

  • Forceville, C., Veale, T., and Feyaerts, K. (2010). Balloonics: The visuals of balloons in comics. In: The Rise and Reason of Comics and Graphic Literature: Critical Essays on the Form, J. Goggin and D. Hassler-Forest (Eds.). Jefferson: McFarland & Company, Inc.Google Scholar

  • Foulsham, T., Wybrow, D., and Cohn, N. (2016). Reading without words: eye movements in the comprehension of comic strips. Applied Cognitive Psychology, 30:566–579.Google Scholar

  • Gernsbacher, M. A., Varner, K. R., and Faust, M. (1990). Investigating differences in general comprehension skill. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16:430–445.Google Scholar

  • Goldberg, W. (2010). The manga phenomenon in America. In: Manga: An Anthology of Global and Cultural Perspectives, T. Johnson-Woods (Ed.). New York: Continuum Books.Google Scholar

  • Goldin-Meadow, S. (2006). Talking and thinking with our hands. Current Directions in Psychological Science, 15:34–39.Google Scholar

  • Goldin-Meadow, S., Mcneill, D., and Singleton, J. (1996). Silence is liberating: removing the handcuffs on grammatical expression in the manual modality. Psychological Review, 103:34–55.CrossrefGoogle Scholar

  • Green, J. (2014). Drawn from the Ground: Sound, Sign and Inscription in Central Australian Sand Stories. Cambridge, UK: Cambridge University Press.Google Scholar

  • Guynes, S. A. (2014). Four-color sound: a peircean semiotics of comic book onomatopoeia. The Public Journal of Semiotics, 6:58–72.Google Scholar

  • Hatfield, C., Heer, J., and Worcester, K. (2013). The Superhero Reader. Jackson: University Press of Mississippi.Google Scholar

  • Kaan, E. (2007). Event-related potentials and language processing: a brief overview. Language and Linguistics Compass, 1:571–591.Google Scholar

  • Lacassin, F. (1972). The comic strip and film language. Film Quarterly, 26:11–23.CrossrefGoogle Scholar

  • Magliano, J. P., Larson, A. M., Higgs, K., and Loschky, L. C. (2015). The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension. Memory & Cognition 44(2):207–219.CrossrefGoogle Scholar

  • Magliano, J. P., Miller, J., and Zwaan, R. A. (2001). Indexing space and time in film understanding. Applied Cognitive Psychology, 15:533–545.Google Scholar

  • Magliano, J. P., and Zacks, J. M. (2011). The impact of continuity editing in narrative film on event segmentation. Cognitive Science, 35:1489–1517.CrossrefGoogle Scholar

  • Martinec, R., and Salway, A. (2005). A system for image–text relations in new (and old) media. Visual Communication, 4:337–371.CrossrefGoogle Scholar

  • Mazur, D., and Danner, A. (2014). Comics: A Global History, 1968 to the Present. London: Thames & Hudson.Google Scholar

  • McCloud, S. (1993). Understanding Comics: The Invisible Art. New York, NY: Harper Collins.Google Scholar

  • McCloud, S. 1996. Understanding manga. Wizard Magazine, 56, 44–48.Google Scholar

  • Moore, S. (2003). In the old days, it woulda been eight pages. In: A Thousand Flowers: Comics, Pop Culture and the World Outside, Brady, M. (Ed.).Google Scholar

  • Neff, W. A. (1977). The Pictorial and Linguistic Features of Comic Book Formulas. Doctoral Doctoral Dissertation, University of Denver.Google Scholar

  • Osaka, M., Yaoi, K., Minamoto, T., and Osaka, N. (2014). Serial changes of humor comprehension for four-frame comic Manga: An fMRI study. Scientific Reports:4.Google Scholar

  • Painter, C., Martin, J. R., and Unsworth, L. (2012). Reading Visual Narratives: Image Analysis of Children’s Picture Books. London: Equi-nox.Google Scholar

  • Pederson, K., and Cohn, N. (2016). The changing pages of comics: Page layouts across eight decades of American superhero comics. Studies in Comics, 7:7–28.Google Scholar

  • Petersen, R. S. (2011). Comics, Manga, and Graphic Novels: A History of Graphic Narratives. Santa Barbara, CA: ABC-CLIO.Google Scholar

  • Pratha, N. K., Avunjian, N., and Cohn, N. (2016). Pow, punch, pika, and chu: The structure of sound effects in genres of American comics and Japanese manga. Multimodal Communication, 5:93–109.Google Scholar

  • Royce, T. D. (2007). Intersemiotic complementarity: a framework for multimodal discourse analysis. In: New Directions in the Analysis of Multimodal Discourse, T. D. Royce, and W. L. Bowcher (Eds.), 63–109. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar

  • Schwan, S., and Ildirar, S. (2010). Watching film for the first time: how adult viewers interpret perceptual discontinuities in film. Psychological Science, 21:970–976.CrossrefGoogle Scholar

  • Stainbrook, E. J. (2016). A little cohesion between friends; or, we’re just exploring our textuality: reconciling cohesion in written language and visual language. In: The Visual Narrative Reader, N. Cohn (Ed.). London: Bloomsbury.Google Scholar

  • Tasić, M., and Stamenković, D. (2015). The interplay of words and images in expressing multimodal metaphors in comics. Procedia – Social and Behavioral Sciences, 212:117–122.Google Scholar

  • Verano, F. (2006). Spectacular consumption: visuality, production, and the consumption of the comics page. International Journal of Comic Art, 8:378–387.Google Scholar

  • Wilkins, D. P. (1997/2016). Alternative representations of space: arrernte narratives in sand. In: The Visual Narrative Reader, N. Cohn (Ed.). London: Bloomsbury.Google Scholar

  • Zwaan, R. A., and Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123:162–185.CrossrefGoogle Scholar

About the article

Published Online: 2017-04-20

Published in Print: 2017-05-24


Citation Information: Multimodal Communication, ISSN (Online) 2230-6587, ISSN (Print) 2230-6579, DOI: https://doi.org/10.1515/mc-2017-0003.

Export Citation

© 2017 Walter de Gruyter GmbH, Berlin/Boston. Copyright Clearance Center

Comments (0)

Please log in or register to comment.
Log in