A Picture is Worth More Words Over Time: Multimodality and Narrative Structure Across Eight Decades of American Superhero Comics

Neil Cohn 1 , Ryan Taylor 1  and Kaitlin Pederson 1
  • 1 Tilburg center for Cognition and Communication, Tilburg University, 5000 LE, Tilburg, Netherlands
Neil Cohn
  • Corresponding author
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LE, Tilburg, Netherlands
  • Email
  • Search for other articles:
  • degruyter.comGoogle Scholar
, Ryan Taylor
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LE, Tilburg, Netherlands
  • Search for other articles:
  • degruyter.comGoogle Scholar
and Kaitlin Pederson
  • Tilburg center for Cognition and Communication, Tilburg University, 5000 LE, Tilburg, Netherlands
  • Search for other articles:
  • degruyter.comGoogle Scholar

Abstract

The visual narratives of comics involve complex multimodal interactions between written language and the visual language of images, where one or the other may guide the meaning and/or narrative structure. We investigated this interaction in a corpus analysis across eight decades of American superhero comics (1940–2010s). No change across publication date was found for multimodal interactions that weighted meaning towards text or across both text and images, where narrative structures were present across images. However, we found an increase over time of narrative sequences with meaning weighted to the visuals, and an increase of sequences without text at all. These changes coincided with an overall reduction in the number of words per panel, a shift towards panel framing with single characters and close-ups rather than whole scenes, and an increase in shifts between temporal states between panels. These findings suggest that storytelling has shifted towards investing more information in the images, along with an increasing complexity and maturity of the visual narrative structures. This has shifted American comics from being textual stories with illustrations to being visual narratives that use text.

  • Abbott, M., and Forceville, C. (2011). Visual representation of emotion in manga: loss of control is loss of hands in Azumanga Daioh Volume 4. Language and Literature, 20:91–112.

    • Crossref
    • Export Citation
  • Bateman, J. A. (2014). Text and Image: A Critical Introduction to the Visual/Verbal Divide. New York: Routledge.

  • Bateman, J. A., and Wildfeuer, J. (2014a). Defining units of analysis for the systematic analysis of comics: a discourse-based approach. Studies in Comics, 5:373–403.

    • Crossref
    • Export Citation
  • Bateman, J. A., and Wildfeuer, J. (2014b). A multimodal discourse theory of visual narrative. Journal of Pragmatics, 74:180–208.

    • Crossref
    • Export Citation
  • Brienza, C. (2016). Manga in America: transnational book publishing and the domestication of Japanese comics. London: Bloomsbury Academic.

  • Burgas, G. (2007). Compressed storytelling versus decompressed storytelling: pros and cons [Online]. Comic Book Resources. Available: http://www.cbr.com/compressed-storytelling-versus-decompressed-storytelling-pros-and-cons/ [Accessed 2/27/2017].

  • Cicci, M. (2015). Turning the page: Fandoms, multimodality, the transformation of the “comic book” superhero. Doctoral Dissertation, Wayne State University.

  • Cohn, N. (2011). A different kind of cultural frame: an analysis of panels in American comics and Japanese manga. Image [&] Narrative, 12:120–134.

  • Cohn, N. (2013a). The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images. London, UK: Bloomsbury.

  • Cohn, N. (2013b). Visual narrative structure. Cognitive Science, 37:413–452.

    • Crossref
    • Export Citation
  • Cohn, N. (2014). The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology, 5:1–9.

  • Cohn, N. (2015). Narrative conjunction’s junction function: the interface of narrative grammar and semantics in sequential images. Journal of Pragmatics, 88:105–132.

    • Crossref
    • Export Citation
  • Cohn, N. (2016). A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition, 146:304–323.

    • Crossref
    • PubMed
    • Export Citation
  • Cohn, N., and Bender, P. (2017). Drawing the line between constituent structure and coherence relations in visual narratives. Journal of Experimental Psychology: Learning, Memory, & Cognition, 43:289–301.

  • Cohn, N., and Ehly, S. (2016). The vocabulary of manga: visual morphology in dialects of Japanese visual language. Journal of Pragmatics, 92:17–29.

    • Crossref
    • Export Citation
  • Cohn, N., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2014). The grammar of visual narrative: neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64:63–70.

    • Crossref
    • PubMed
    • Export Citation
  • Cohn, N., and Kutas, M. (2015). Getting a cue before getting a clue: event-related potentials to inference in visual narrative comprehension. Neuropsychologia, 77:267–278.

    • Crossref
    • PubMed
    • Export Citation
  • Cohn, N., and Kutas, M. (Under Review). What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing.

  • Cohn, N., Paczynski, M., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2012a). (Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension. Cognitive Psychology, 65:1–38.

    • Crossref
    • Export Citation
  • Cohn, N., Taylor-Weiner, A., and Grossman, S. (2012b). Framing attention in Japanese and American comics: cross-cultural differences in attentional structure. Frontiers in Psychology – Cultural Psychology, 3:1–12.

  • Coogan, P. (2006). Superhero: The Secret Origin of a Genre. Austin, TX: MonkeyBrain.

  • Cutting, J. E. (2015). The framing of characters in popular movies. Art & Perception, 3:191–212.

    • Crossref
    • Export Citation
  • Cutting, J. E., Delong, J. E., and Nothelfer, C. E. (2010). Attention and the evolution of hollywood film. Psychological Science, 21:432–439.

    • Crossref
    • PubMed
    • Export Citation
  • Duncan, R., and Smith, M. J. (2009). The Power of Comics. New York: Continuum Books.

  • Duncan, R., Smith, M. J., and Levitz, P. (2015). The Power of Comics. New York: Continuum Books.

  • Forceville, C. (2011). Pictorial runes in Tintin and the Picaros. Journal of Pragmatics, 43:875–890.

    • Crossref
    • Export Citation
  • Forceville, C. (2016). Conceptual metaphor theory, blending theory, and other cognitivist perspectives on comics . In: The Visual Narrative Reader, N. Cohn (Ed.). Bloomsbury: London.

  • Forceville, C., and Urios-Aparisi, E. (2009). Multimodal Metaphor. New York: Mouton De Gruyter.

  • Forceville, C., Veale, T., and Feyaerts, K. (2010). Balloonics: The visuals of balloons in comics. In: The Rise and Reason of Comics and Graphic Literature: Critical Essays on the Form, J. Goggin and D. Hassler-Forest (Eds.). Jefferson: McFarland & Company, Inc.

  • Foulsham, T., Wybrow, D., and Cohn, N. (2016). Reading without words: eye movements in the comprehension of comic strips. Applied Cognitive Psychology, 30:566–579.

    • Crossref
    • Export Citation
  • Gernsbacher, M. A., Varner, K. R., and Faust, M. (1990). Investigating differences in general comprehension skill. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16:430–445.

    • PubMed
    • Export Citation
  • Goldberg, W. (2010). The manga phenomenon in America. In: Manga: An Anthology of Global and Cultural Perspectives, T. Johnson-Woods (Ed.). New York: Continuum Books.

  • Goldin-Meadow, S. (2006). Talking and thinking with our hands. Current Directions in Psychological Science, 15:34–39.

    • Crossref
    • Export Citation
  • Goldin-Meadow, S., Mcneill, D., and Singleton, J. (1996). Silence is liberating: removing the handcuffs on grammatical expression in the manual modality. Psychological Review, 103:34–55.

    • Crossref
    • PubMed
    • Export Citation
  • Green, J. (2014). Drawn from the Ground: Sound, Sign and Inscription in Central Australian Sand Stories. Cambridge, UK: Cambridge University Press.

  • Guynes, S. A. (2014). Four-color sound: a peircean semiotics of comic book onomatopoeia. The Public Journal of Semiotics, 6:58–72.

    • Crossref
    • Export Citation
  • Hatfield, C., Heer, J., and Worcester, K. (2013). The Superhero Reader. Jackson: University Press of Mississippi.

  • Kaan, E. (2007). Event-related potentials and language processing: a brief overview. Language and Linguistics Compass, 1:571–591.

    • Crossref
    • Export Citation
  • Lacassin, F. (1972). The comic strip and film language. Film Quarterly, 26:11–23.

    • Crossref
    • Export Citation
  • Magliano, J. P., Larson, A. M., Higgs, K., and Loschky, L. C. (2015). The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension. Memory & Cognition 44(2):207–219.

  • Magliano, J. P., Miller, J., and Zwaan, R. A. (2001). Indexing space and time in film understanding. Applied Cognitive Psychology, 15:533–545.

    • Crossref
    • Export Citation
  • Magliano, J. P., and Zacks, J. M. (2011). The impact of continuity editing in narrative film on event segmentation. Cognitive Science, 35:1489–1517.

    • Crossref
    • PubMed
    • Export Citation
  • Martinec, R., and Salway, A. (2005). A system for image–text relations in new (and old) media. Visual Communication, 4:337–371.

    • Crossref
    • Export Citation
  • Mazur, D., and Danner, A. (2014). Comics: A Global History, 1968 to the Present. London: Thames & Hudson.

  • McCloud, S. (1993). Understanding Comics: The Invisible Art. New York, NY: Harper Collins.

  • McCloud, S. 1996. Understanding manga. Wizard Magazine, 56, 44–48.

  • Moore, S. (2003). In the old days, it woulda been eight pages. In: A Thousand Flowers: Comics, Pop Culture and the World Outside, Brady, M. (Ed.).

  • Neff, W. A. (1977). The Pictorial and Linguistic Features of Comic Book Formulas. Doctoral Doctoral Dissertation, University of Denver.

  • Osaka, M., Yaoi, K., Minamoto, T., and Osaka, N. (2014). Serial changes of humor comprehension for four-frame comic Manga: An fMRI study. Scientific Reports:4.

  • Painter, C., Martin, J. R., and Unsworth, L. (2012). Reading Visual Narratives: Image Analysis of Children’s Picture Books. London: Equi-nox.

  • Pederson, K., and Cohn, N. (2016). The changing pages of comics: Page layouts across eight decades of American superhero comics. Studies in Comics, 7:7–28.

    • Crossref
    • Export Citation
  • Petersen, R. S. (2011). Comics, Manga, and Graphic Novels: A History of Graphic Narratives. Santa Barbara, CA: ABC-CLIO.

  • Pratha, N. K., Avunjian, N., and Cohn, N. (2016). Pow, punch, pika, and chu: The structure of sound effects in genres of American comics and Japanese manga. Multimodal Communication, 5:93–109.

  • Royce, T. D. (2007). Intersemiotic complementarity: a framework for multimodal discourse analysis. In: New Directions in the Analysis of Multimodal Discourse, T. D. Royce, and W. L. Bowcher (Eds.), 63–109. Mahwah, NJ: Lawrence Erlbaum Associates.

  • Schwan, S., and Ildirar, S. (2010). Watching film for the first time: how adult viewers interpret perceptual discontinuities in film. Psychological Science, 21:970–976.

    • Crossref
    • PubMed
    • Export Citation
  • Stainbrook, E. J. (2016). A little cohesion between friends; or, we’re just exploring our textuality: reconciling cohesion in written language and visual language. In: The Visual Narrative Reader, N. Cohn (Ed.). London: Bloomsbury.

  • Tasić, M., and Stamenković, D. (2015). The interplay of words and images in expressing multimodal metaphors in comics. Procedia – Social and Behavioral Sciences, 212:117–122.

    • Crossref
    • Export Citation
  • Verano, F. (2006). Spectacular consumption: visuality, production, and the consumption of the comics page. International Journal of Comic Art, 8:378–387.

  • Wilkins, D. P. (1997/2016). Alternative representations of space: arrernte narratives in sand. In: The Visual Narrative Reader, N. Cohn (Ed.). London: Bloomsbury.

  • Zwaan, R. A., and Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123:162–185.

    • Crossref
    • PubMed
    • Export Citation
Purchase article
Get instant unlimited access to the article.
$42.00
Log in
Already have access? Please log in.


or
Log in with your institution

Journal + Issues

Multimodal Communication is a venue for researchers in diverse fields of study, who explore varied & multimodal ways to conduct research & illustrate findings. It is open to papers from anthropology to art, math, psychology, science, & beyond. Articles share a commitment to developing multimodality and may use visuals/writing/sound to explore a theme; be highly theoretical or may be based on an empirical study integrating a number of modes.

Search