The visual narratives of comics involve complex multimodal interactions between written language and the visual language of images, where one or the other may guide the meaning and/or narrative structure. We investigated this interaction in a corpus analysis across eight decades of American superhero comics (1940–2010s). No change across publication date was found for multimodal interactions that weighted meaning towards text or across both text and images, where narrative structures were present across images. However, we found an increase over time of narrative sequences with meaning weighted to the visuals, and an increase of sequences without text at all. These changes coincided with an overall reduction in the number of words per panel, a shift towards panel framing with single characters and close-ups rather than whole scenes, and an increase in shifts between temporal states between panels. These findings suggest that storytelling has shifted towards investing more information in the images, along with an increasing complexity and maturity of the visual narrative structures. This has shifted American comics from being textual stories with illustrations to being visual narratives that use text.
Abbott, M., and Forceville, C. (2011). Visual representation of emotion in manga: loss of control is loss of hands in Azumanga Daioh Volume 4. Language and Literature, 20:91–112.10.1177/0963947011402182)| false
Cohn, N. (2015). Narrative conjunction’s junction function: the interface of narrative grammar and semantics in sequential images. Journal of Pragmatics, 88:105–132.10.1016/j.pragma.2015.09.001)| false
Cohn, N., and Bender, P. (2017). Drawing the line between constituent structure and coherence relations in visual narratives. Journal of Experimental Psychology: Learning, Memory, & Cognition, 43:289–301.
Cohn, N., and Ehly, S. (2016). The vocabulary of manga: visual morphology in dialects of Japanese visual language. Journal of Pragmatics, 92:17–29.
Cohn, N., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2014). The grammar of visual narrative: neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64:63–70.
Cohn, N., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2014). The grammar of visual narrative: neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64:63–70.10.1016/j.neuropsychologia.2014.09.01825241329)| false
Cohn, N., and Kutas, M. (2015). Getting a cue before getting a clue: event-related potentials to inference in visual narrative comprehension. Neuropsychologia, 77:267–278.2632070610.1016/j.neuropsychologia.2015.08.026)| false
Cohn, N., and Kutas, M. (Under Review). What’s your neural function, visual narrative conjunction? Grammar, meaning, and fluency in sequential image processing.
Cohn, N., Paczynski, M., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2012a). (Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension. Cognitive Psychology, 65:1–38.
Cohn, N., Paczynski, M., Jackendoff, R., Holcomb, P. J., and Kuperberg, G. R. (2012a). (Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension. Cognitive Psychology, 65:1–38.10.1016/j.cogpsych.2012.01.003)| false
Cohn, N., Taylor-Weiner, A., and Grossman, S. (2012b). Framing attention in Japanese and American comics: cross-cultural differences in attentional structure. Frontiers in Psychology – Cultural Psychology, 3:1–12.
Coogan, P. (2006). Superhero: The Secret Origin of a Genre. Austin, TX: MonkeyBrain.
Cutting, J. E. (2015). The framing of characters in popular movies. Art & Perception, 3:191–212.
Forceville, C. (2016). Conceptual metaphor theory, blending theory, and other cognitivist perspectives on comics . In: The Visual Narrative Reader, N. Cohn (Ed.). Bloomsbury: London.
Forceville, C., and Urios-Aparisi, E. (2009). Multimodal Metaphor. New York: Mouton De Gruyter.
Forceville, C., Veale, T., and Feyaerts, K. (2010). Balloonics: The visuals of balloons in comics. In: The Rise and Reason of Comics and Graphic Literature: Critical Essays on the Form, J. Goggin and D. Hassler-Forest (Eds.). Jefferson: McFarland & Company, Inc.
Foulsham, T., Wybrow, D., and Cohn, N. (2016). Reading without words: eye movements in the comprehension of comic strips. Applied Cognitive Psychology, 30:566–579.
Gernsbacher, M. A., Varner, K. R., and Faust, M. (1990). Investigating differences in general comprehension skill. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16:430–445.2140402)| false
Goldin-Meadow, S., Mcneill, D., and Singleton, J. (1996). Silence is liberating: removing the handcuffs on grammatical expression in the manual modality. Psychological Review, 103:34–55.10.1037/0033-295X.103.1.348650298)| false
Magliano, J. P., Larson, A. M., Higgs, K., and Loschky, L. C. (2015). The relative roles of visuospatial and linguistic working memory systems in generating inferences during visual narrative comprehension. Memory & Cognition 44(2):207–219.
Magliano, J. P., Miller, J., and Zwaan, R. A. (2001). Indexing space and time in film understanding. Applied Cognitive Psychology, 15:533–545.
Petersen, R. S. (2011). Comics, Manga, and Graphic Novels: A History of Graphic Narratives. Santa Barbara, CA: ABC-CLIO.
Pratha, N. K., Avunjian, N., and Cohn, N. (2016). Pow, punch, pika, and chu: The structure of sound effects in genres of American comics and Japanese manga. Multimodal Communication, 5:93–109.
Royce, T. D. (2007). Intersemiotic complementarity: a framework for multimodal discourse analysis. In: New Directions in the Analysis of Multimodal Discourse, T. D. Royce, and W. L. Bowcher (Eds.), 63–109. Mahwah, NJ: Lawrence Erlbaum Associates.
Schwan, S., and Ildirar, S. (2010). Watching film for the first time: how adult viewers interpret perceptual discontinuities in film. Psychological Science, 21:970–976.
Schwan, S., and Ildirar, S. (2010). Watching film for the first time: how adult viewers interpret perceptual discontinuities in film. Psychological Science, 21:970–976.10.1177/095679761037263220530390)| false
Stainbrook, E. J. (2016). A little cohesion between friends; or, we’re just exploring our textuality: reconciling cohesion in written language and visual language. In: The Visual Narrative Reader, N. Cohn (Ed.). London: Bloomsbury.
Tasić, M., and Stamenković, D. (2015). The interplay of words and images in expressing multimodal metaphors in comics. Procedia – Social and Behavioral Sciences, 212:117–122.
Tasić, M., and Stamenković, D. (2015). The interplay of words and images in expressing multimodal metaphors in comics. Procedia – Social and Behavioral Sciences, 212:117–122.10.1016/j.sbspro.2015.11.308)| false
Multimodal Communication is a venue for researchers in diverse fields of study, who explore varied & multimodal ways to conduct research & illustrate findings. It is open to papers from anthropology to art, math, psychology, science, & beyond. Articles share a commitment to developing multimodality and may use visuals/writing/sound to explore a theme; be highly theoretical or may be based on an empirical study integrating a number of modes.