Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter March 9, 2011

Making music with images: interactive audiovisual performance systems for the deaf

Michael S. Grierson
From the journal


This paper describes the technical and aesthetic approach ­utilised for the development of an interactive audiovisual performance system designed specifically for use by children with multiple learning difficulties, including deafness and autism. Sound is transformed in real-time through the implementation of a Fast Fourier Transform and translated into a moving image. This image is adapted so that relevant information can be understood and manipulated visually in real-time. Finally, the image is turned back into sound with only minimal delay. The translation process is based on research in computer music, neuroscience, perception and abstract film studies. The system is achieving a significant user base amongst the deaf and disabled community through projects involving the British Broadcasting Corporation and the London Philharmonic Orchestra. Alongside continuing collaboration with the Sonic Arts Network, the system has now been made freely available to the public, and is being developed as a speech therapy tool.

Corresponding author: Michael S. Grierson, Director of Creative Computing, Goldsmiths Department of Computing, Goldsmiths, University of London, New Cross, London, SE14 6NW, UK

Received: 2010-11-1
Accepted: 2010-12-1
Published Online: 2011-03-09
Published in Print: 2011-3-1

©2011 by Walter de Gruyter Berlin New York

Scroll Up Arrow