Multimodal digital semiotics: the interaction of language with other resources

  • 1 Multimodal Analysis Lab, Interactive & Digital Media Institute (IDMI), National University of Singapore, 9 Prince George's Park, Singapore 118408


A “multimodal digital semiotics” approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News. Prototype software for multimodal analysis and network visualizations are used in combination with Allen's interval algebra to model multimodal semantic data and identify significant patterns based on temporal relationships between different semiotic choices. The resulting semantic patterns reveal combinations of semiotic choices that were drawn upon by the different speakers during the interview, accounting for how the climate denialist emerges as the dominant and convincing interviewee. Such an approach combines the objectivity of mathematical modeling and scientific visualization with the interpretive power of Halliday's systemic theory.

Purchase article
Get instant unlimited access to the article.
Log in
Already have access? Please log in.

Journal + Issues

Text & Talk (founded as TEXT in 1981) is an internationally recognized forum for interdisciplinary research in language, discourse, and communication studies, focusing, among other things, on the situational and historical nature of text/talk production; the cognitive and sociocultural processes of language practice/action; and participant-based structures of meaning negotiation and multimodal alignment.