Jump to ContentJump to Main Navigation
Show Summary Details
In This Section

Multimodal Communication

Ed. by Norris, Sigrid

2 Issues per year

Online
ISSN
2230-6587
See all formats and pricing
In This Section

A Picture is Worth More Words Over Time: Multimodality and Narrative Structure Across Eight Decades of American Superhero Comics

Neil Cohn
  • Corresponding author
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LETilburg, Netherlands
  • Email:
/ Ryan Taylor
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LETilburg, Netherlands
/ Kaitlin Pederson
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LETilburg, Netherlands
Published Online: 2017-04-20 | DOI: https://doi.org/10.1515/mc-2017-0003

Abstract

The visual narratives of comics involve complex multimodal interactions between written language and the visual language of images, where one or the other may guide the meaning and/or narrative structure. We investigated this interaction in a corpus analysis across eight decades of American superhero comics (1940–2010s). No change across publication date was found for multimodal interactions that weighted meaning towards text or across both text and images, where narrative structures were present across images. However, we found an increase over time of narrative sequences with meaning weighted to the visuals, and an increase of sequences without text at all. These changes coincided with an overall reduction in the number of words per panel, a shift towards panel framing with single characters and close-ups rather than whole scenes, and an increase in shifts between temporal states between panels. These findings suggest that storytelling has shifted towards investing more information in the images, along with an increasing complexity and maturity of the visual narrative structures. This has shifted American comics from being textual stories with illustrations to being visual narratives that use text.

Keywords: visual language; multimodality; narrative structure; comics; discourse

References

  • Abbott, M., and Forceville, C. (2011). Visual representation of emotion in manga: loss of control is loss of hands in Azumanga Daioh Volume 4. Language and Literature, 20:91–112.

  • Bateman, J. A. (2014). Text and Image: A Critical Introduction to the Visual/Verbal Divide. New York: Routledge.

  • Bateman, J. A., and Wildfeuer, J. (2014a). Defining units of analysis for the systematic analysis of comics: a discourse-based approach. Studies in Comics, 5:373–403.

  • Bateman, J. A., and Wildfeuer, J. (2014b). A multimodal discourse theory of visual narrative. Journal of Pragmatics, 74:180–208. [Crossref]

  • Brienza, C. (2016). Manga in America: transnational book publishing and the domestication of Japanese comics. London: Bloomsbury Academic.

  • Burgas, G. (2007). Compressed storytelling versus decompressed storytelling: pros and cons [Online]. Comic Book Resources. Available: http://www.cbr.com/compressed-storytelling-versus-decompressed-storytelling-pros-and-cons/ [Accessed 2/27/2017].

  • Cicci, M. (2015). Turning the page: Fandoms, multimodality, the transformation of the “comic book” superhero. Doctoral Dissertation, Wayne State University.

  • Cohn, N. (2011). A different kind of cultural frame: an analysis of panels in American comics and Japanese manga. Image [&] Narrative, 12:120–134.

  • Cohn, N. (2013a). The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. London, UK: Bloomsbury.

  • Cohn, N. (2013b). Visual narrative structure. Cognitive Science, 37:413–452. [Crossref]

  • Cohn, N. (2014). The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology, 5:1–9. [Crossref]

  • Cohn, N. (2015). Narrative conjunction’s junction function: the interface of narrative grammar and semantics in sequential images. Journal of Pragmatics, 88:105–132. [Crossref]

  • Cohn, N. (2016). A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition, 146:304–323. [Crossref]

  • Cohn, N., and Bender, P. (2017). Drawing the line between constituent structure and coherence relations in visual narratives. Journal of Experimental Psychology: Learning, Memory, & Cognition, 43:289–301.

  • Cohn, N., and Ehly, S. (2016). The vocabulary of manga: visual morphology in dialects of Japanese visual language. Journal of Pragmatics, 92:17–29. [Crossref]

  • Cohn, N., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2014). The grammar of visual narrative: neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64:63–70. [Crossref]

  • Cohn, N., and Kutas, M. (2015). Getting a cue before getting a clue: event-related potentials to inference in visual narrative comprehension. Neuropsychologia, 77:267–278. [Crossref]

  • Cohn, N., and Kutas, M. (Under Review). What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing.

  • Cohn, N., Paczynski, M., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2012a). (Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension. Cognitive Psychology, 65:1–38. [Crossref]

  • Cohn, N., Taylor-Weiner, A., and Grossman, S. (2012b). Framing attention in Japanese and American comics: cross-cultural differences in attentional structure. Frontiers in Psychology – Cultural Psychology, 3:1–12.

  • Coogan, P. (2006). Superhero: The Secret Origin of a Genre. Austin, TX: MonkeyBrain.

  • Cutting, J. E. (2015). The framing of characters in popular movies. Art & Perception, 3:191–212. [Crossref]

  • Cutting, J. E., Delong, J. E., and Nothelfer, C. E. (2010). Attention and the evolution of hollywood film. Psychological Science, 21:432–439. [Crossref]

  • Duncan, R., and Smith, M. J. (2009). The Power of Comics. New York: Continuum Books.

  • Duncan, R., Smith, M. J., and Levitz, P. (2015). The Power of Comics. New York: Continuum Books.

  • Forceville, C. (2011). Pictorial runes in Tintin and the Picaros. Journal of Pragmatics, 43:875–890. [Crossref]

  • Forceville, C. (2016). Conceptual metaphor theory, blending theory, and other cognitivist perspectives on comics . In: The Visual Narrative Reader, N. Cohn (Ed.). Bloomsbury: London.

  • Forceville, C., and Urios-Aparisi, E. (2009). Multimodal Metaphor. New York: Mouton De Gruyter.

  • Forceville, C., Veale, T., and Feyaerts, K. (2010). Balloonics: The visuals of balloons in comics. In: The Rise and Reason of Comics and Graphic Literature: Critical Essays on the Form, J. Goggin and D. Hassler-Forest (Eds.). Jefferson: McFarland & Company, Inc.

  • Foulsham, T., Wybrow, D., and Cohn, N. (2016). Reading without words: eye movements in the comprehension of comic strips. Applied Cognitive Psychology, 30:566–579.

  • Gernsbacher, M. A., Varner, K. R., and Faust, M. (1990). Investigating differences in general comprehension skill. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16:430–445.

  • Goldberg, W. (2010). The manga phenomenon in America. In: Manga: An Anthology of Global and Cultural Perspectives, T. Johnson-Woods (Ed.). New York: Continuum Books.

  • Goldin-Meadow, S. (2006). Talking and thinking with our hands. Current Directions in Psychological Science, 15:34–39.

  • Goldin-Meadow, S., Mcneill, D., and Singleton, J. (1996). Silence is liberating: removing the handcuffs on grammatical expression in the manual modality. Psychological Review, 103:34–55. [Crossref]

  • Green, J. (2014). Drawn from the Ground: Sound, Sign and Inscription in Central Australian Sand Stories. Cambridge, UK: Cambridge University Press.

  • Guynes, S. A. (2014). Four-color sound: a peircean semiotics of comic book onomatopoeia. The Public Journal of Semiotics, 6:58–72.

  • Hatfield, C., Heer, J., and Worcester, K. (2013). The Superhero Reader. Jackson: University Press of Mississippi.

  • Kaan, E. (2007). Event-related potentials and language processing: a brief overview. Language and Linguistics Compass, 1:571–591.

  • Lacassin, F. (1972). The comic strip and film language. Film Quarterly, 26:11–23. [Crossref]

  • Magliano, J. P., Larson, A. M., Higgs, K., and Loschky, L. C. (2015). The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension. Memory & Cognition 44(2):207–219. [Crossref]

  • Magliano, J. P., Miller, J., and Zwaan, R. A. (2001). Indexing space and time in film understanding. Applied Cognitive Psychology, 15:533–545.

  • Magliano, J. P., and Zacks, J. M. (2011). The impact of continuity editing in narrative film on event segmentation. Cognitive Science, 35:1489–1517. [Crossref]

  • Martinec, R., and Salway, A. (2005). A system for image–text relations in new (and old) media. Visual Communication, 4:337–371. [Crossref]

  • Mazur, D., and Danner, A. (2014). Comics: A Global History, 1968 to the Present. London: Thames & Hudson.

  • McCloud, S. (1993). Understanding Comics: The Invisible Art. New York, NY: Harper Collins.

  • McCloud, S. 1996. Understanding manga. Wizard Magazine, 56, 44–48.

  • Moore, S. (2003). In the old days, it woulda been eight pages. In: A Thousand Flowers: Comics, Pop Culture and the World Outside, Brady, M. (Ed.).

  • Neff, W. A. (1977). The Pictorial and Linguistic Features of Comic Book Formulas. Doctoral Doctoral Dissertation, University of Denver.

  • Osaka, M., Yaoi, K., Minamoto, T., and Osaka, N. (2014). Serial changes of humor comprehension for four-frame comic Manga: An fMRI study. Scientific Reports:4.

  • Painter, C., Martin, J. R., and Unsworth, L. (2012). Reading Visual Narratives: Image Analysis of Children’s Picture Books. London: Equi-nox.

  • Pederson, K., and Cohn, N. (2016). The changing pages of comics: Page layouts across eight decades of American superhero comics. Studies in Comics, 7:7–28.

  • Petersen, R. S. (2011). Comics, Manga, and Graphic Novels: A History of Graphic Narratives. Santa Barbara, CA: ABC-CLIO.

  • Pratha, N. K., Avunjian, N., and Cohn, N. (2016). Pow, punch, pika, and chu: The structure of sound effects in genres of American comics and Japanese manga. Multimodal Communication, 5:93–109.

  • Royce, T. D. (2007). Intersemiotic complementarity: a framework for multimodal discourse analysis. In: New Directions in the Analysis of Multimodal Discourse, T. D. Royce, and W. L. Bowcher (Eds.), 63–109. Mahwah, NJ: Lawrence Erlbaum Associates.

  • Schwan, S., and Ildirar, S. (2010). Watching film for the first time: how adult viewers interpret perceptual discontinuities in film. Psychological Science, 21:970–976. [Crossref]

  • Stainbrook, E. J. (2016). A little cohesion between friends; or, we’re just exploring our textuality: reconciling cohesion in written language and visual language. In: The Visual Narrative Reader, N. Cohn (Ed.). London: Bloomsbury.

  • Tasić, M., and Stamenković, D. (2015). The interplay of words and images in expressing multimodal metaphors in comics. Procedia – Social and Behavioral Sciences, 212:117–122.

  • Verano, F. (2006). Spectacular consumption: visuality, production, and the consumption of the comics page. International Journal of Comic Art, 8:378–387.

  • Wilkins, D. P. (1997/2016). Alternative representations of space: arrernte narratives in sand. In: The Visual Narrative Reader, N. Cohn (Ed.). London: Bloomsbury.

  • Zwaan, R. A., and Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123:162–185. [Crossref]

About the article

Published Online: 2017-04-20

Published in Print: 2017-05-24



Citation Information: Multimodal Communication, ISSN (Online) 2230-6587, ISSN (Print) 2230-6579, DOI: https://doi.org/10.1515/mc-2017-0003. Export Citation

Comments (0)

Please log in or register to comment.
Log in