Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Text & Talk

An Interdisciplinary Journal of Language, Discourse & Communication Studies

Ed. by Sarangi, Srikant

6 Issues per year


IMPACT FACTOR 2017: 0.426
5-year IMPACT FACTOR: 0.724

CiteScore 2017: 0.63

SCImago Journal Rank (SJR) 2017: 0.326
Source Normalized Impact per Paper (SNIP) 2017: 0.625

Online
ISSN
1860-7349
See all formats and pricing
More options …
Volume 33, Issue 4-5

Issues

Multimodal digital semiotics: the interaction of language with other resources

Kay L. O'Halloran
  • Corresponding author
  • Multimodal Analysis Lab, Interactive & Digital Media Institute (IDMI), National University of Singapore, 9 Prince George's Park, Singapore 118408
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Marissa K. L. E / Alexey Podlasov / Sabine Tan
Published Online: 2013-08-27 | DOI: https://doi.org/10.1515/text-2013-0030

Abstract

A “multimodal digital semiotics” approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News. Prototype software for multimodal analysis and network visualizations are used in combination with Allen's interval algebra to model multimodal semantic data and identify significant patterns based on temporal relationships between different semiotic choices. The resulting semantic patterns reveal combinations of semiotic choices that were drawn upon by the different speakers during the interview, accounting for how the climate denialist emerges as the dominant and convincing interviewee. Such an approach combines the objectivity of mathematical modeling and scientific visualization with the interpretive power of Halliday's systemic theory.

Keywords: multimodal digital semiotics; mathematical modeling; multimodality; systemic functional theory

About the article

Kay L. O'Halloran

Kay O'Halloran is Director of the Multimodal Analysis Lab, Deputy Director of the Interactive & Digital Media Institute, and Associate Professor in the Department of English Language and Literature at the National University of Singapore. Her areas of research include multimodal analysis, social semiotics, mathematics discourse, and the development of interactive digital media technologies and mathematical and scientific visualization techniques for multimodal and sociocultural analytics. She is Principal Investigator for large research projects in these areas in the Multimodal Analysis Lab at the National University of Singapore and has published widely in these fields. Address for correspondence: Multimodal Analysis Lab, Interactive & Digital Media Institute (IDMI), National University of Singapore, 9 Prince George's Park, Singapore 118408, Singapore 〈 〉.

Marissa K. L. E

Marissa K. L. E is Research Associate at the Multimodal Analysis Lab in the Interactive & Digital Media Institute at the National University of Singapore. Her interests lie in systemic functional linguistics, multimodal discourse analysis, and their applications in analyzing the constitution, perpetuation, and evolution of ideologies and culture as manifested in traditional and emergent forms of media communication. She has worked on interdisciplinary projects involving the application of mathematical modeling techniques to the analysis of multimodal data and the development of interactive and collaborative software for multimodal analysis for research and educational purposes. Her publications reflect this synthesis of digital and traditional humanities in various areas such as climate science news, institutional business news, corporate advertisements, and education. Address for correspondence: 〈 〉.

Alexey Podlasov

Alexey Podlasov worked as Post-Doctoral Research Fellow at the Multimodal Analysis Lab in the Interactive and Digital Media Institute (IDMI) at the National University of Singapore from 2008 to 2012. His background is in computer science and applied mathematics with research interests in data modeling, visualization and analytics, computer vision, image and video processing, as well as with software prototyping and development. He is currently working with photorealistic rendering techniques for mobile 3D graphics. Address for correspondence: 〈 〉.

Sabine Tan

Sabine Tan is Research Associate at the Multimodal Analysis Lab, Interactive & Digital Media Institute, at the National University of Singapore. Her research interests include multimodal discourse analysis, social semiotics, and visual communication. She has published on topics that include the application of multidisciplinary perspectives within social semiotic theory to the analysis of institutional discourses involving traditional and new media, such as business news mediated on the Internet, corporate television advertisements, corporate Web pages, and other emergent multimodal discourse genres. She has worked on interdisciplinary projects involving the development of interactive and collaborative software for the multimodal analysis of images and video texts for research and educational purposes. Address for correspondence: 〈 〉.


Published Online: 2013-08-27

Published in Print: 2013-08-19


Citation Information: Text & Talk, Volume 33, Issue 4-5, Pages 665–690, ISSN (Online) 1860-7349, ISSN (Print) 1860-7330, DOI: https://doi.org/10.1515/text-2013-0030.

Export Citation

©[2013] by Walter de Gruyter Berlin Boston.Get Permission

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

[1]
Kay L. O’Halloran, Sabine Tan, Duc-Son Pham, John Bateman, and Andrew Vande Moere
Journal of Mixed Methods Research, 2018, Volume 12, Number 1, Page 11

Comments (0)

Please log in or register to comment.
Log in