AutisMitr: Emotion Recognition Assistive Tool for Autistic Children

Akansha Singh 1  and Surbhi Dewan 2
  • 1 Department of Computer Science & Engineering, ASET, Amity University, Noida, India
  • 2 The NorthCap University, , Gurgaon, India


Assistive technology has proven to be one of the most significant inventions to aid people with Autism to improve the quality of their lives. In this study, a real-time emotion recognition system for autistic children has been developed. Emotion recognition is implemented by executing three stages: Face identification, Facial Feature extraction, and feature classification. The objective is to frame a system that includes all three stages of emotion recognition activity that executes expeditiously in real time. Thus, Affectiva SDK is implemented in the application. The propound system detects at most 7 facial emotions: anger, disgust, fear, joy, sadness, contempt, and surprise. The purpose for performing this study is to teach emotions to individuals suffering from autism, as they lack the ability to respond appropriately to others emotions. The proposed application was tested with a group of typical children aged 6–14 years, and positive outcomes were achieved.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] Kanner, L. (1943). Autistic disturbances of affective contact. Nervous child, 2(3), 217-250.

  • [2] American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub.

  • [3] Baio, J. (2014). Prevalence of autism spectrum disorder among children aged 8 years-autism and developmental disabilities monitoring network, 11 sites, United States, 2010.

  • [4] Baron-Cohen, S., & Wheelwright, S. (2004). The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. Journal of autism and developmental disorders, 34(2), 163-175.

  • [5] Magdin, M., & Prikler, F. (2018). Real time facial expression recognition using webcam and SDK affectiva. IJIMAI, 5(1), 7-15.

  • [6] Capps, L., Kasari, C., Yirmiya, N., & Sigman, M. (1993). Parental perception of emotional expressiveness in children with autism. Journal of Consulting and Clinical Psychology, 61(3), 475.

  • [7] McDuff, D., Mahmoud, A., Mavadati, M., Amr, M., Turcot, J., & Kaliouby, R. E. (2016, May). AFFDEX SDK: a cross-platform real-time multi-face expression recognition toolkit. In Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems (pp. 3723-3726). ACM.

  • [8] Mazefsky, C. A., Herrington, J., Siegel, M., Scarpa, A., Maddox, B. B., Scahill, L., & White, S. W. (2013). The role of emotion regulation in autism spectrum disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 52(7), 679-688.

  • [9] Lopresti, E. F., Bodine, C., & Lewis, C. (2008). Assistive technology for cognition [Understanding the Needs of Persons with Disabilities. IEEE Engineering in Medicine and Biology Magazine, 27(2), 29-39.

  • [10] Knight, V., McKissick, B. R., & Saunders, A. (2013). A review of technology-based interventions to teach academic skills to students with autism spectrum disorder. Journal of autism and developmental disorders, 43(11), 2628-2648.

  • [11] Aresti-Bartolome, N., & Garcia-Zapirain, B. (2014). Technologies as support tools for persons with autistic spectrum disorder: a systematic review. International journal of environmental research and public health, 11(8), 7767-7802.

  • [12] Blasco, S., Cerro, P., Elena, M., & Uceda, J. D. (2009). Autism and Technology: an approach to new technology-based therapeutic tools. In World Congress on Medical Physics and Biomedical Engineering, September 7-12, 2009, Munich, Germany (pp. 340-343). Springer, Berlin, Heidelberg.

  • [13] Mirenda, P. (2003). Toward functional augmentative and alternative communication for students with autism. Language, speech, and hearing services in schools.

  • [14] Liddle, K. (2001). Implementing the picture exchange communication system (PECS). International journal of language & communication disorders, 36(S1), 391-395.

  • [15] Cortiella, C., & Horowitz, S. H. (2014). The state of learning disabilities: Facts, trends and emerging issues. New York: National Center for Learning Disabilities, 25.

  • [16] Hayes, G. R., Hirano, S., Marcu, G., Monibi, M., Nguyen, D. H., & Yeganyan, M. (2010). Interactive visual supports for children with autism. Personal and ubiquitous computing, 14(7), 663-680.

  • [17] Noor, H. A. M., Shahbodin, F., & Pee, N. C. (2012). Serious game for autism children: review of literature. World Academy of Science, Engineering and Technology, 64(124), 647-652.

  • [18] Rahman, M. M., Ferdous, S. M., & Ahmed, S. I. (2010, December). Increasing intelligibility in the speech of the autistic children by an interactive computer game. In 2010 IEEE International Symposium on Multimedia (pp. 383-387). IEEE.

  • [19] Alqahtani, A., Jaafar, N., & Alfadda, N. (2011, October). Interactive speech based games for autistic children with asperger syndrome. In The 2011 International Conference and Workshop on Current Trends in Information Technology (CTIT 11) (pp. 126-131). IEEE.

  • [20] Anwar, A., Rahman, M. M., Ferdous, S. M., Anik, S. A., & Ahmed, S. I. (2011, July). A computer game based approach for increasing fluency in the speech of the autistic children. In 2011 IEEE 11th International Conference on Advanced Learning Technologies (pp. 17-18). IEEE.

  • [21] Hassan, A. Z., Zahed, B. T., Zohora, F. T., Moosa, J. M., Salam, T., Rahman, M. M., ... & Ahmed, S. I. (2011, December). Developing the concept of money by interactive computer games for autistic children. In 2011 IEEE International Symposium on Multimedia (pp. 559-564). IEEE.

  • [22] Gates, K. (2004). The past perfect promise of facial recognition technology. ACDIS Occasional Paper.

  • [23] Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2003). Face recognition: A literature survey. ACM computing surveys (CSUR), 35(4), 399-458.

  • [24] Baron-Cohen, S., Golan, O., & Ashwin, E. (2009). Can emotion recognition be taught to children with autism spectrum conditions?. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3567-3574.

  • [25] Cheng, L., Kimberly, G., & Orlich, F. (2002). KidTalk: online therapy for Asperger’s syndrome. Microsoft Research.

  • [26] Goldsmith, T. R., & LeBlanc, L. A. (2004). Use of technology in interventions for children with autism. Journal of Early and Intensive Behavior Intervention, 1(2), 166.

  • [27] Dautenhahn, K., & Werry, I. (2004). Towards interactive robots in autism therapy: Background, motivation and challenges. Pragmatics & Cognition, 12(1), 1-35.

  • [28] Robins, B., Dautenhahn, K., & Dickerson, P. (2009, February). From isolation to communication: a case study evaluation of robot assisted play for children with autism with a minimally expressive humanoid robot. In 2009 Second International Conferences on Advances in Computer-Human Interactions(pp. 205-211). IEEE.

  • [29] El Kaliouby, R., & Robinson, P. (2005). The emotional hearing aid: an assistive tool for children with Asperger syndrome. Universal Access in the Information Society, 4(2), 121-134.

  • [30] Ekman, P., & Friesen, W. V. (1978). Manual for the facial action coding system. Consulting Psychologists Press.

  • [31] Pantic, M., & Rothkrantz, L. J. (2000). Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis & Machine Intelligence, (12), 1424-1445.

  • [32] Barrett, L. F., Mesquita, B., Ochsner, K. N., & Gross, J. J. (2007). The experience of emotion. Annu. Rev. Psychol., 58, 373-403.

  • [33] Magdin, M., Benko, Ľ., & Koprda, Š. (2019). A Case Study of Facial Emotion Classification Using Affdex. Sensors, 19(9), 2140.

  • [34] Batty, M., & Taylor, M. J. (2003). Early processing of the six basic facial emotional expressions. Cognitive Brain Research, 17(3), 613-620.

  • [35] Leony, D., Muñoz-Merino, P. J., Pardo, A., & Kloos, C. D. (2013). Provision of awareness of learners’ emotions through visualizations in a computer interaction-based environment. Expert Systems with Applications, 40(13), 5093-5100.

  • [36] Wells, L. J., Gillespie, S. M., & Rotshtein, P. (2016). Identification of emotional facial expressions: Effects of expression, intensity, and sex on eye gaze. PloS one, 11(12), e0168307.

  • [37] Bartlett, M. S., Viola, P. A., Sejnowski, T. J., Golomb, B. A., Larsen, J., Hager, J. C., & Ekman, P. (1996). Classifying facial action. In Advances in neural information processing systems (pp. 823-829).

  • [38] Wong, B., Cronin-Golomb, A., & Neargarder, S. (2005). Patterns of visual scanning as predictors of emotion identification in normal aging. Neuropsychology, 19(6), 739.

  • [39] Jensen, O. H. (2008). Implementing the Viola-Jones face detection algorithm (Master’s thesis, Technical University of Denmark, DTU, DK-2800 Kgs. Lyngby, Denmark).

  • [40] Dalal, N., & Triggs, B. (2005, June). Histograms of oriented gradients for human detection. In international Conference on computer vision & Pattern Recognition (CVPR’05) (Vol. 1, pp. 886-893). IEEE Computer Society.

  • [41] Magdin, M., & Prikler, F. (2018). Real Time Facial Expression Recognition Using Webcam and SDK Affectiva. International Journal of Interactive Multimedia & Artificial Intelligence, 5(1).

  • [42] McDuff, D., Mahmoud, A., Mavadati, M., Amr, M., Turcot, J., & Kaliouby, R. E. (2016, May). AFFDEX SDK: a cross-platform real-time multi-face expression recognition toolkit. In Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems (pp. 3723-3726). ACM.


Journal + Issues

Open Computer Science is an open access, peer-reviewed journal. The journal publishes research results in the following fields: algorithms and complexity theory, artificial intelligence, bioinformatics, networking and security systems,
programming languages, system and software engineering, and theoretical foundations of computer science.