Alireza Taheri, Azadeh Shariati, Rozita Heidari, Mojtaba Shahab, Minoo Alemi and Ali Meghdari

Impacts of using a social robot to teach music to children with low-functioning autism

Open Access
De Gruyter Open Access | Published online: April 28, 2021

Abstract

This article endeavors to present the impact of conducting robot-assisted music-based intervention sessions for children with low-functioning (LF) autism. To this end, a drum/xylophone playing robot is used to teach basic concepts of how to play the instruments to four participants with LF autism during nine educational sessions. The main findings of this study are compared to similar studies conducted with children with high-functioning autism. Our main findings indicated that the stereotyped behaviors of all the subjects decreased during the course of the program with an approximate large Cohen’s d effect size. Moreover, the children showed some improvement in imitation, joint attention, and social skills from the Pre-Test to Post-Test. In addition, regarding music education, we indicated that while the children could not pass a test on the music notes or reading music phrases items because of their cognitive deficits, they showed acceptable improvements (with a large Cohen’s d effect size) in the Stambak Rhythm Reproduction Test, which means that some rhythm learning occurred for the LF participants. In addition, we indicated that parenting stress levels decreased during the program. This study presents some potential possibilities of performing robot-assisted interventions for children with LF autism.

1 Introduction

Music lessons are an effective tool to engage children with autism spectrum disorder (ASD) in non-verbal and rhythmic communications [1]. It is reported in ref. [2] that at least 12% of worldwide interventions for individuals with autism have included music-based games in the rehabilitation/treatment process. Ample evidence which demonstrates that playing or even teaching music in intervention sessions could significantly increase the impact of the rehabilitation process for children with autism [3,4]. In ref. [3], the authors emphasized that early behavioral clinical programs are very important for children with ASD, and music therapy can be an effective treatment program. In ref. [4], the authors studied how the perception of musical patterns incorporated in applied behavior analysis verbal behavior (ABA VB) operants impacted the production of speech in 22 (3 to 5-year-old) participants with autism. They reported that both speech and music exercises were beneficial for the production of the ABA verbal operants, and they concluded that music incorporated ABA VB training was most effective in echoic production. Certain previous studies have inspired other scientists to facilitate the multi-system development of individuals with special needs by using an embodied music-based method/approach [2]. Music-based tasks involved in these intervention sessions included playing pre-recorded files and/or teachers/therapists playing musical instruments [2,5]. Kalas [6] and Kim et al. [7] reported that the eye-contact, turn taking, and Joint Attention (JA) skills of the participants with ASD improved through their active music-based sessions. References [810] indicate that conducting music-based intervention sessions led to a decrease in self-injuries and stereotyped behaviors in their subjects with autism. In addition, it has been shown that music-based intervention sessions can improve gestural and verbal communication [11], social and emotional skills [12,13,14], and behaviors [15] of children with autism. According to ref. [2], a serious gap in this field of study is the lack of investigations on the impact of music-based treatment protocols on gross/fine motor skills of individuals with autism.

The use of social robots in the education and cognitive rehabilitation of children with autism has been widely investigated during the last two decades [16,17,18,19,20]. Some review papers have discussed, summarized, and argued [21,22,23,24] the findings of different studies in this field of research. The study in ref. [21] mentions that engagement and attention levels of individuals with autism seem to improve during robot-assisted intervention sessions as well as novel social behaviors (such as spontaneous imitation and JA) elicited from participants with ASD. The mentioned article presents an overview of the physical appearance, complexity, and degrees of freedom of robots used in autism therapy. In this regard, the authors concluded that there is no clear-cut agreement on the preferred shape of robots in this area and that humanoid-robots, animal-like robots, and even cartoon-based robots ones have shown a variety of positive impacts on children with ASD in intervention sessions. Moreover, in ref. [21], readers can find comparisons of different human–robot interactions (HRI) and various roles of robots in the field of autism that are focused on encouraging imitative behaviors, eliciting JA, and mediating turn-taking behaviors between the participants and human therapists. All in all, ref. [21] presents an overall positive view of using robots as potential tools in autism rehabilitation. Cabibihan et al. [22] separated the role of robots in autism interventions into friendly playmates, agents for eliciting behaviors, social mediators, social actors, and/or personal therapists. They mentioned that robots are less complex and intimidating than humans for participants with ASD, which made robot-assisted sessions more effective for them. The authors emphasized that future experiments should be designed across different research groups. Another article [23] reviewed more than 400 papers analyzing the use of social robots in autism therapy/rehabilitation. Along with the promising results reported for robot-assisted interventions for children with autism, the authors of ref. [23] emphasized three serious gaps in this field including diversity in the focus of the studies, contribution towards autism impairments, and effectiveness of the interventions. The authors in ref. [24] present a critical review of robot-assisted interventions for children with ASD considering users’ responses to robots as well as the use of robots in eliciting behaviors, modeling skills, and providing appropriate feedback. They indicated that most of the reported results were exploratory with serious limitations, which means they do not necessarily provide strong evidence for generalization or advocate the broad use of robots in autism clinics. So far, most robot-assisted interventions have been performed/published for children with high-functioning (HF) autism [21,23], and unfortunately, the study and comparison of such technological treatment protocols for children with low-functioning (LF) autism has received much less attention [25,26]. It should be noted that on the autism spectrum, individuals with autism who are or appear to be closer to normal/typically developing people are considered to be HF. In general, the ability to communicate verbally and the amount of verbal speech, and/or the awareness of social conventions for individuals with HF autism are higher than peers with LF autism [26,27], while the deficits of children with LF autism are often more visually and aurally obvious to the casual observer.[1] Working with children with LF autism is very challenging; they may show aggression and other difficult behaviors (due to the combined effects of linguistic/cognitive impairments) [27]. However, the mentioned distinctions are not absolute, and individuals with autism have individual strengths and needs and may behave completely different in various situations [8,26,27].

Considering the mentioned potential of (1) music education/therapy and (2) robot-assisted interventions for children with ASD, in this study we have combined these two paradigms (again) to present an educational-therapeutic environment that can provide a variety of educational and socio-cognitive situations/tasks for participants with autism. To this end, this article endeavors to present the results of running robot-assisted music-based intervention sessions with an emphasis on children with LF ASD. Therefore, an attempt is made to investigate the impacts of teaching music to four LF subjects via a drum/xylophone player NAO robot with regard to (1) music education, (2) social and cognitive skills, and (3) parenting stress levels of the participants during a case study. The current research is built on our previous studies using a social robot to systematically teach music to children with autism [8]. While the focus of our previous study was on children with HF autism, in this article, all of the recruited participants have been diagnosed with LF ASD, which will most likely make the educational interventions more challenging for the teachers. It should be noted that children with ASD, especially LF children, have deficits that usually do not allow complex brain connectivity and flexibility. This motived us to study children with LF autism to ascertain the following: (1) a more in-depth understanding of the unknowns and potentials of LF children’s behaviors during HRI since almost all of the previous works on LF participants with robots have been exploratory [26,27,28,29,30], (2) a better understanding of LF impairments and the differences on the autism spectrum to provide the necessary information to design/propose appropriate robot-assisted intervention protocols, and (3) to study whether the positive changes happening/reported for the LF participant (and/or HF subjects) in ref. [8] could be generalized to other children with LF autism. As previously mentioned, there is ample evidence that this type of intervention is successful when applied by a human therapist; however, it is necessary to note that while the robot plays as an attractive assistive tool for human-therapists to make the education/rehabilitation process more appealing in such studies, in our experiments any unpredictable situations during the sessions were handled by the human teacher. In other words, as the study protocol is described throughout this manuscript, we prefer to have both (1) the knowledge, experience, and influence of the teacher, as well as (2) the social robot’s potential in any education/therapeutic process for music activities (Robots have been shown to be able to facilitate children’s learning process from different aspects [such as music education/therapy area], by providing interactive, spontaneous, consistent, and repetitive trainings for users [8,21,29,30]). In ref. [31], the authors tried to find a preliminary answer to the following question: “Who is a better teacher for children with low-functioning autism, human-mediators or robots?” After conducting their study with 23 LF autism participants, they reported that when the provided tasks are highly structured, it does not matter who (i.e. human or robot) serves as the teacher for the subjects.

Conducting robot-assisted intervention sessions to systematically teach music to children with ASD has some rehabilitation potential. However, to the best of our knowledge, the impact of using social robots in such classes (especially robots that play musical instruments) on the educational and cognitive skills of participants with autism is covered less in the literature [8,30,32,33,34,35]. In ref. [30], a NAO robot was programmed to play a drum to provide consistent repetitive training for rhythm education for participants with ASD and Typically Developing subjects. The main findings of this work confirmed the potential of such studies in improvement in JA skills, conversation bouts, and interlimb/intralimb coordination of the subjects. In another study [32], a xylophone-player Darwin robot was used to conduct a set of imitation tasks during a 7-week program for two children with ASD. While the authors did not observe any significant progress in the participants’ imitation skills, they did indicate some improvement in the communication and concentration of the children with autism. In ref. [33], the authors investigated the importance of robot-assisted music-based sessions on the social attention patterns of 36 participants with autism by conducting a set of scenarios (e.g. rhythm, robotic, and comparison interventions). The role of the robot was to involve the children in signing, making requests, and some cognitive tasks. It should be noted that no musical instrument was played by their NAO robot. In ref. [35], the authors reviewed 13 papers (including both robot- and non-robot-assisted studies) that addressed music, dance, and movement-based interventions for children with autism. One of the studied themes reported that music with instruments can improve fine motor skills, social skills, and bilateral hand coordination in participants with ASD. In ref. [8], we proposed a novel hierarchical robot-assisted music-based protocol to teach the fundamentals of music and playing a drum/xylophone to individuals with ASD. After conducting the interventions for three children with HF, and one child with LF autism, we found interesting progress regarding the rhythm and music notes education, as well as some improvements in social and communication skills of the HF subjects. Our previous study inspired us to conduct the current research. Due to the obvious difference in the learning rate and performance of the LF subject in moderate/high level music tasks compared to the participants with HF autism in ref. [8], we estimated that the designed robot-assisted music-based schedule was too cumbersome for the participant with LF autism. Therefore, we proposed that one possible way to make such classes more effective for children with LF ASD would be to slow down the teaching process (as in the current study) and/or increase the number or density of the interventions.

1.1 Research questions

The preliminary exploratory research questions of this study are as follows: (Q1) Does a teacher-assistant humanoid robot have the ability to teach music rhythms and notes to children with LF autism? (Q2) Does a robot-assisted music-based education have an impact on the social and cognitive skills of children with LF autism? and (Q3) Can robot-assisted interventions affect the stress levels of the LF participants’ parents during music-education classes? In addition, this article will also calculate the effect sizes between the findings of this study and ref. [8] (i.e. a comparison of the impact of the program on LF and HF participants). We will document how many of the previously reported improvements for participants with LF or HF autism in ref. [8] reoccur for the LF subjects of this study.

2 Methodology

2.1 Participants

Four children with LF ASD, including three boys and one girl, participated in this study. For the participants’ recruitment, after getting the research permit and ethical approval of the study from the Iran University of Medical Sciences, a flyer was sent to the Center for the Treatment of Autistic Disorders (CTAD), Iran. It should be noted that a “Subjects’ Consent letter” and “Test Information and the Subjects’ Rights” were presented to the volunteers. All of the children and parents voluntarily took part in this study and signed a pledge/consent form to maintain high moral standards. Their mean age and standard deviation were 5.75 and 0.75 years, respectively. The children are identified as R1 to R4 throughout the article. More information about the participants is presented in Table 1. It should be noted that none of the participants had any prior musical education, and they had no experience with any robot-assisted interventions before this program. All of the participants had verbal deficits, which made handling the classes very challenging for the human teacher. All subjects were diagnosed with LF ASD by the doctors and psychologists of the CTAD. They have received some Applied Behavioral Analysis (ABA) clinical interventions before this research. Based on the medical records (i.e. GARS questionnaire) of the participants, the initial scores of their autism severity were 92, 80, 83, and 93, respectively, which indicate severe autism of the subjects.

Table 1

The children’s details

No. ID Gender Age Autism severity
1 R1 Female 5 years and 2 months LF autism, with verbal deficits
2 R2 Male 5 years and 1 month LF autism, with ADHD and verbal deficits
3 R3 Male 6 years and 4 months LF autism, poor verbal skills (i.e. the mean length of utterance less than 2), and lack of eye-contact
4 R4 Male 6 years and 5 months LF autism, with verbal deficits, stubborn and resistant to education

2.2 The robot and musical instruments

We used and programmed a NAO H-21 robot[2] to play a drum and a xylophone. We chose the Iranian male name “Nima” for the robot in our program. Two mallets were attached to Nima’s hands during the intervention sessions. Nima’s voice commands were pre-recorded (in the Persian language) and uploaded to the robot. The role of the robot in our study was to be a facilitating tool and a teacher assistant in music classes. Because of the LF participants’ cognitive deficits, low instruction perception, maladaptive behaviors, wandering around the room, etc., the human teacher was highly engaged and intervened directly in the educational sessions.

2.3 Experimental setup and intervention protocol

This study was conducted at the Social and Cognitive Robotics Lab at Sharif University of Technology, Iran in the presence of the child, the Nima robot, one human teacher, and a robot operator in a Wizard of Oz style control. The room size was approximately 3.5 × 4.5 × 3 m3. The schematics of the intervention room is presented in Figure 1. While it is more common to separate the examination and observation areas in such studies, we unfortunately did not have access to a better/bigger room to conduct our intervention sessions. Nonetheless, based on our previous experience in working with children with autism during robot-assisted interventions, we realized that children less than 8 years-old (especially those with noticeable cognitive deficits such as LF subjects) do not usually understand that the robot is operated by a human operator and do not pay (even moderate) attention to the operator during the sessions. Based on our observations during this study, we did not note any participant distractions because of the operator’s attendance during the sessions. The program included one session of Baseline (Week #1), a Pre-Test (Week #6 before starting the first music intervention session), nine robot-assisted music-based educational sessions (Weeks #6–14), a Post-Test (Week #14 after the last music session), and a Follow-up Test (Week #22, 2 months after the last session). Each intervention session took about 20–30 min.

Figure 1 The schematic of the intervention room and its main components.

Figure 1

The schematic of the intervention room and its main components.

During the sessions, the participants attempted to perform the robot/teacher’s instructions in a set of imitation, JA, and turn-taking music-based games. The designed music games/tasks are presented in Table 2. All of the pre-set/structured music activities contained systematic and hierarchical music tasks, including an orientation/introduction session, striking the drum/xylophone bars using the mallets, matching the played colors, hitting the bars with and without counting, rhythm perception, working memory games, learning notes, and playing musical phrases. In the virtual xylophone game (i.e. Game #10), each child stands in front of the Kinect sensor and observes eight colored music bars (corresponding to our xylophone) on the screen. The subject should hit the appropriate sequence of bars per the teacher/robot’s instructions with a colored object or mallet (e.g. in this study, we gave a green hat as the mallet to the children and asked them to play the determined notes). The protocol prompt design is shown in Figure 2. Considering that the LF children’s behaviors were often unpredictable during the sessions (e.g. wandering around the game room, having maladaptive behaviors, hitting Nima, inattention to the robot/teacher, etc.), all of the robot’s commands (as well as the selected exercises with the appropriate difficulty levels) were sent to Nima by the robot operator during the sessions. It should be noted that the criteria for passing an exercise and moving on to the next harder task was to correctly play that exercise for at least 50% of the performed time.

Table 2

The music-based robot-assisted interventions schedule

Hierarchical tasks Outline Short description of task modes Goals and potential impacts
1 Orientation session a Introduction of the robot to the participants – Familiarize the subjects to the class environment and the Nima robot
Robot’s performance includes greeting, singing, dancing, etc.
2 Strike musical instruments using mallets a No bars are removed from the instrument (Easy mode) – Learn how to make sounds with the xylophone and drum – Imitation – Visual pursuit – Attention
After each robot’s strike, participant should hit any arbitrary bar of the xylophone
b Some of the bars are removed from the xylophone (Difficult mode). Repeat 2a with fewer bars on the instrument
c Repeat 2a while the participant uses both of his/her hands in order to play the xylophone and drum
d Repeat 2b while the participant uses both of his/her hands in order to play the xylophone and drum
3 Match colors during the music tasks a Considering bars’ colors, the participant should imitate Nima’s strike – Color recognition – Imitation – Instruction perception – Joint attention
b Nima strikes a xylophone bar and the participant should point to the rectangle with the same color on the screen (and vice versa)
c Participants should find and hit the same color on the instrument as Nima’s eyes’ LED color
4 Hit the instruments and counting a Participants’ strikes imitate those of the Nima robot considering the number/order of the played notes. The participant and Nima count “one-two” while hitting the instruments – Perception of the numbers and counting – Verbal short-term memory
b Repeat 4a while saying “Baam–Baam” instead of “one-two”
5 Hit the bars without counting a The subject should imitate the notes Nima played without counting – Imitation – Auditory discrimination – Complex tasks’ instructions
6 Rhythm perception a Nima strikes 3 to 5 notes on the instruments in order with different rhythms. The participant should imitate the robot considering the order and the time interval between the played notes – Auditory imitation skill – Visual pursuit – Hand –eye coordination
7 Working memory game a The participant should remember the colors and play them on the xylophone as shown/said by Nima – Working memory – Joint attention – Dual task performance
8 Teach/read the music notes a Nima teaches the music notes (i.e. Do, Re, Me, etc.) and then shows some cards to the participant. The subject should say the name of the notes and strike the equivalent bar on the instrument – Teach the music notes – Shifting attention skill – Auditory memory
9 Playing music phrases a Participant plays short music phrases while saying the notes’ names. Nima assesses the subject’s performance, i.e. playing preliminary/easy songs. Parents can also attend the session and see their child’s performance – Cooperation between behavior and motor skills – Auditory memory – Play music phrases
10 Playing the virtual xylophone and TURN-taking games a Nima hits certain bars. The subject should imitate the same notes on the virtual xylophone (and vice versa) – Sharing attention between different tasks – Turn-taking – Involve the child in Child–Robot –Teacher games – Generalization skills
b The teacher or parent hits the real/virtual bars. The participant and the robot imitate the same notes on the virtual and real xylophone, respectively. (It is expected that the subject generalizes the learnt music skills on the virtual xylophone)
Figure 2 The structure of the triadic Robot–Child–Teacher in the program.

Figure 2

The structure of the triadic Robot–Child–Teacher in the program.

It should be mentioned that because of the LF children’s deficits in instruction perception and cognitive skills, the “learning notes” and “playing musical phrases” items (i.e. Games #8 and #9) could not be systematically performed for our subjects, and we jumped to Game #10 (i.e. playing the virtual xylophone) in the last session. The LF childrens’ imitation of the robot performing the tasks during the educational sessions was also significantly weaker than the HF participants in ref. [8] in all of the schedule items (which is to be expected), even in the striking two note tasks. It should be noted that (correctly) imitating the Nima robot striking three or more notes in order (while considering the time interval between each note) was often hard/challenging even for the HF subjects, and the participants of this study were rarely able to pass such items (e.g. with appropriate counting). Figure 3 shows snapshots of playing the real and virtual xylophones in the intervention sessions.

Figure 3 Snapshots of three of the participants playing the real and virtual xylophones in the intervention sessions: (a) considering bars’ colors, the child should imitate Nima’s strikes (i.e. Game #3-a). (b) The child is imitating the robot’s strikes on the virtual xylophone (i.e. Game #10-a). (c) The child did not do the tasks appropriately based on the instructions of playing the virtual xylophone (i.e. Game #10-a).

Figure 3

Snapshots of three of the participants playing the real and virtual xylophones in the intervention sessions: (a) considering bars’ colors, the child should imitate Nima’s strikes (i.e. Game #3-a). (b) The child is imitating the robot’s strikes on the virtual xylophone (i.e. Game #10-a). (c) The child did not do the tasks appropriately based on the instructions of playing the virtual xylophone (i.e. Game #10-a).

2.4 Assessment tools

Two kinds of assessments were conducted to find preliminary exploratory answers to the research questions of this study (Figure 4). Firstly, the Developmental Assessments, which included assessing (1) the imitation and JA skills of the children by extracting and performing 30 tasks from the Early Social Communication Scales (ESCS) [36,37] and regular imitation exercises in autism clinics, as well as (2) assessing the rhythm reproduction ability of the subjects by a human therapist performing the Stambak Rhythmic Structures Reproduction Test [38] three times in the Baseline, Pre-Test, and Post-Test. Regarding the imitation assessments, the subject should perform approximately 15 gross and fine motor imitation exercises presented by the teacher, including the gross/fine imitation of whole body movements and several imitation activities with toys and objects (such as the opposite arm–leg task, balloon game, arranging the objects, match the ribbons, the car game, matches game, etc.; Figure 4a). The percentage of successful performances is considered as the imitation score of each participant. To assess JA, different objects/tasks were presented to the children based on the ESCS instruction to obtain the number of eliciting responses (i.e. Responding to Joint Attention [RJA]) and initiations (both automatic initiation and per the teacher’s requests) to the provided situations (i.e. Initiating Joint Attention [IJA]) with the human-mediator. Again, the percentages of the successful performances for each subject are considered as the RJA and IJA scores, respectively.

Figure 4 Snapshots of the assessment sessions: (a) ESCS/imitation test and (b) Stambak test.

Figure 4

Snapshots of the assessment sessions: (a) ESCS/imitation test and (b) Stambak test.

Secondly, four Questionnaires, including the Autism Social Skills Profile (ASSP) [39], Gilliam Autism Rating Scale (GARS) [40], Autism Checklist [5], and Parenting Stress Index-Short Form (PSI-SF) [41], were filled in by the parents four times during the Baseline, Pre-Test, Post-Test, and Follow-up Test sessions. The assessment sessions (i.e. the Baseline, Pre-Test, Post-Test, and Follow-up Test) were conducted without the presence of the Nima robot.

In this study, the Cohen’s d effect size (which is independent of the study’s sample size) [42] between the Pre- and Post-Tests on the questionnaires is reported in Section 4. The Cohen’s d effect size may not represent an actual improvement in the children’s abilities, and it is not presented to make a strong claim or generalize the findings in this study. However, based on the limited available data (which makes it impossible to conduct statistically significant tests), the Cohen’s d effect size can be used to reveal the potential of the designed music-based program’s effects on the participants with LF autism. Moreover, taking inspiration from ref. [43], we also calculated and reported the Pearson correlation coefficients for the cases where a significant association existed between the developmental assessments and questionnaires’ results in Section 4.

3 Qualitative observations of participants’ performances during interventions

After performing the qualitative video analysis of the sessions, the important descriptive observations and recorded progress trends of our four participants are reported in this section. Moreover, a review of the results for the LF subject in ref. [8] is also presented for an initial comparison with the provided qualitative analysis of the sessions of this study.

R1: She had low visual attention in Sessions 1–3. She knew the colors and recognized the robot’s eye color correctly. She interacted with Nima under the guidance of the teacher in the first three sessions. One-hit imitation tasks were performed correctly by the child on both the xylophone and the drum. The sequence of using her hands gradually improved. When playing the instruments, it was easier for her to start her hits with her right hand. R1 did not do badly in imitating the rhythm on the drums when she paid enough attention to the tasks. Imitation of simultaneous strikes by two mallets on two xylophone bars spaced more than one note apart was commonly mistaken. R1 was not very confident in performing two-hit tasks on the xylophone. Usually, the duration of the child’s attention was short during the sessions. In Sessions 4–6, the human teacher reported that her eye contact increased in comparison to Sessions 1–3. The child was helped by the human-mediator in performing two-hit tasks with both hands in order. She imitated the rhythms on the drum under the direct guidance of the teacher and understood the counting technique better than saying “Baam–Baam” when performing the tasks. The child’s attention and cooperation somewhat increased in Sessions 4–6. She voluntarily communicated with the robot and performed the robot’s exercises without much interference from the teacher. She did not perform well in the tasks that she had to start with her left hand. In 4-hit tasks on the drum, she could only perform the task via counting and only with one of her hands (preferably the right hand). In Sessions 7–9, it was easier for her to communicate with Nima than before. The number of consecutive exercises she participated in was short and the child was often distracted from the music tasks. A short musical phrase in the form of some consecutive colors was given to the child; she performed very well in recognizing the order of the notes on the xylophone. However, her performance was not good in terms of quality and consistency of playing the notes. On the virtual xylophone, she realized the connection between the hat and the image. R1 did the exercises very well with appropriate speed (Figure 2b). In the end, it seems that the child has acquired the basic skills needed to learn music and can gradually imitate simple melodic sentences on the xylophone. One of the most important observations was the impact of the sessions on the child in terms of increased interaction and communication (with the teacher and the robot) in contrast to the first two sessions when she avoided even looking at the robot.

R2: The child did not cooperate in Sessions 1 and 2 and did not follow the teacher’s instructions. He was interested in the task in which the robot eye color changed but did not cooperate in other exercises. On occasions, he refused to cooperate with any of the methods of encouragement, and it was very difficult for the teacher to control him. In Sessions 4–6, R2 did some of the exercises presented by the robot. The teacher understood that when the child performed the movements in the standing position, his cooperation increased slightly. This child paid attention and was attracted to each exercise for about five minutes; after that, R2 got bored, and the exercise had to be changed. He also performed worse in some of the intermediate sessions (e.g. Sessions 4–6) than in his first performances. Essentially, he showed resistance to all educational classes. R2 repeated repetitive and meaningless words (related to movies he had seen) to himself during the exercises. In Sessions 7–9, his cooperation increased a bit, and he played some notes on the xylophone, but he still showed a lot of resistance and was not compliant. The incentives that seemed to be useful for others did not affect his performance. He liked the virtual xylophone very much and realized the related exercises; however, he did not cooperate and did not do the tasks based on the instructions. Instead of playing the notes via the colored hat, he put it on and just looked at the virtual bars on the screen (Figure 2c). At the conclusion of the interventions, R2 was only able to perform the one-hit and two-hit exercises on the xylophone and drum to a good extent and hit the relevant note with the guidance of the robot. Overall, the participant was unable to show significant progress during these sessions due to his low functioning autism and consistent lack of cooperation.

R3: In Sessions 1–3, he was very excited and ran around the room. It was very difficult to control the child’s movements. The robot distracted the child in the first two sessions. R3 had no control over the mallets’ hits on the xylophone and hit randomly. He even showed maladaptive behaviors by hitting the mallets on the body of the instrument. With the guidance of the teacher and food incentive, he could only correctly imitate a few one-hit imitation movements on the xylophone when most of the bars were removed from the xylophone. R3 did not know all of the colors (such as purple). In Sessions 4–6, he began to hit the correct bars and no longer hit the body of the instrument. R3 could understand the exercises. In hitting only one bar, he could tell where Nima had hit the xylophone when all of the bars existed on the instrument. The order of using hands for hitting the drum improved compared to the first sessions. Incentives had a significant impact on his performance; however, they made it more difficult to control the child’s behavior. Having stereotyped behavior, he sometimes laughed a lot for no reason. Mirror imitation with the guidance of the human mediator was relatively good. The one-hit imitation on the xylophone was great. R3 could only perform the two-hit imitation with the help of the teacher and a very small number of bars. He usually did it by the trial and error method. In Sessions 4–6, while he was not yet able to identify the bars, he could detect the range positions of the bars played by the robot. He had become curious about the robot. In Sessions 7–9, he liked the virtual xylophone very much. R3 realized the instruction of the virtual game very quickly and used the colored hat to play the virtual instrument well. He recognized the spatial direction of the hand movement well. Sometimes he showed maladaptive behaviors by throwing the hat around. Verbal–motor coordination obviously improved for the child. It was great when he performed two hits correctly without the guidance and control of the human mediator. R3 was able to perform up to three hits on the xylophone and four hits on the drum with the help of counting, although he sometimes made mistakes in the order of the movements. He could control the power when hitting with the mallets. His cooperation increased when he reacted appropriately to the robot’s greetings. He was occasionally not patient enough to wait until the performance of the robot was complete.

R4: In Sessions 1–3, the subject was easily distracted. R4 did not have the desire to interact with the robot himself; but, he interacted with the robot under the guidance of the teacher. He was able to recognize the robot’s eye color properly and played the matching notes on the xylophone. At the beginning (i.e. Sessions 1 and 2), R4 did not use the correct hand to play the music; however, after a little practice, he was able to use both his right and left hands appropriately. He was able to hit the single note on the xylophone correctly with the help of the teacher; however, he had serious difficulties in the two-hit exercises. By performing some movements on the child’s body, the human mediator gradually guided him and improved his performance in playing the notes. After a few sessions, he imitated two-hit tasks with a single mallet correctly. For this subject, persuasive encounters worked better than verbal incentives. In Sessions 4–6, he had a better relationship with Nima and the teacher’s involvement gradually reduced. R4 understood the exercises correctly and performed his tasks independently, and his concentration increased when performing the tasks. He implemented mirror imitation little by little with the help of the teacher and he was able to then perform these tasks independently. The child’s verbal–motor coordination in performing the exercises was still weak at this level. However, he imitated two-hit tasks correctly. In Sessions 7–9, he understood how the virtual xylophone worked during a complex task. However, he could not initially figure out how to use the hat to play the notes. The human mediator believed that verbal–motor coordination at the end of the program was relatively at the proper level for R4. He played up to 3-hit tasks correctly and held the mallets appropriately. R4 had good cooperation for playing the virtual xylophone, and he was able to independently play a short melody on the virtual xylophone. Overall, the child’s performance in Sessions 7–9 seemed to be significantly improved.

Table 3 summarizes the main difficulties and improvements of the subjects during the program. In summary of this section, in both this study and ref. [8], we indicated that the proposed intervention was too cumbersome for the LF subjects. Moreover, expecting participants with LF autism to use both hands consecutively to play the xylophone and the drum as well as learning the notes systematically (not observed in any of our subjects) seems to be a set of very difficult goals, especially in less than 10 sessions. To learn 3-hit tasks on the xylophone using both hands in order and 4-hit tasks on the drum while counting seems to be more logical expectations/goals for these children in similar music-based programs.

Table 3

The main qualitative difficulties of the participants in the first half and their main improvements in the second half of the program

No. Participant’s ID Main difficulties in the first half of the sessions Main improvements in the second half of the sessions
1 R1 – Low visual attention – Weak imitation of strikes by two mallets on two bars spaced more than one note apart – Inappropriate performance in the tasks which started with the left hand – Slight improvement in using both hands in order – Eye contact and communication with the robot
2 R2 – Serious resistance to educational classes – Low cooperation in the sitting position – No significant progress in music learning
3 R3 – No control over the mallets’ hits on the xylophone – Improvement in cooperation with the robot – Appropriate reactions to the robot’s greetings
4 R4 – No desire to interact with the robot himself – Verbal motor coordination – Independently played a short melody

4 Results

In this section, the results of the developmental assessments, including the Stambak Rhythm Test, the ESCS/Imitation, and the questionnaires, are presented. The discussion of the results and their comparison to other works are presented in the next section. We also present a comparison of this study’s findings with our previous paper, conducted with children with HF autism [8], considering the interventions’ effect size (Table 4).

Table 4

Cohen’s d effect size between the Pre-Test and Post-Test scores of the questionnaires (the results of ref. [8] are also presented in the table for comparison of the impacts of the program on LF and HF subjects)

Questionnaire Subscale LF participants HF participants in ref. [8]
ASSP Overall score 0.62 0.47
Social interaction 1.25 0.25
Social participation/avoidance 0.16 1.23
Detrimental social behaviors 0.07 0.72
Autism checklist Overall score 0.90 Not assessed
Verbal, language, and communication 0.76
Socializing 0.27
Sensory or cognitive awareness 0.31
Physical and behavioral health 0.48
GARS Overall score 0.19 0.76
Stereotyped behaviors 0.65 1.44
Communication 0.13 1.23
Social interaction 0.20 0.36
PSI-SF Overall score 0.75 0.47

The Cohen’s d effect sizes around or greater than 0.8 are in bold.

4.1 Developmental assessments

The results of the developmental assessment tests in the Baseline, Pre-Test, and Post-Test are shown in Figure 5a–d.

Figure 5 The results of the developmental assessments sessions for the LF subjects in the Baseline (Week #1), Pre-Test (Week #6), and the Post-Test (Week #14): (a) Stambak Rhythm Test, (b) imitation success percentage scores, (c) RJA scores, and (d) IJA scores.

Figure 5

The results of the developmental assessments sessions for the LF subjects in the Baseline (Week #1), Pre-Test (Week #6), and the Post-Test (Week #14): (a) Stambak Rhythm Test, (b) imitation success percentage scores, (c) RJA scores, and (d) IJA scores.

4.1.1 Stambak rhythm test

Figure 5a shows the results of the Stambak Rhythm Test in the Baseline, Pre-Test, and Post-Test. This test contains 21 easy to hard level musical rhythmic actions that were performed on the drum to assess the children regarding their music education.

Figure 5a shows that all of the LF participants showed some improvement, even though slight, in rhythm recognition/reproduction from the Pre-Test to Post-Test, which shows that some educational process occurred regarding the rhythm reproduction. However, none of the LF subjects achieved the musical notes phrase item, nor the reading music phrase item in this study. The calculated Cohen’s d effect size regarding the Stambak Test for our LF participants is 1.1, which is considered to be large (especially due to the noticeable improvement recorded in R4’s performance). The lower scores of the LF participants in comparison to the HF subjects in ref. [8] in this test are most likely due to the lower cognitive ability of the participants. It should be noted that R3 and R4’s zero scores in the Baseline are because of their lack of cooperation with the human teacher in the Stambak Test. This issue is briefly discussed in Section 5.

In contrast to ref. [8], the reading music phrases test could not be conducted by the human teacher to assess their learnt music knowledge because our LF subjects (unfortunately) could not complete all the scheduled items due to low cognitive skills.

4.1.2 Assessing the social and cognitive skills of the participants

To assess the social and cognitive skills of the subjects, based on the ESCS and common imitation tasks conducted for children with ASD in ABA interventions, 30 tasks were selected/performed three times (i.e. Baseline, Pre-Test, and Post-Test) for the participants by a child psychologist. The conducted items included instruction perception, turn-taking, social reciprocity, gaze-shifting, JA, and gross and fine imitation skills of the participants. Then, after video coding and movie analysis of the developmental assessment sessions, the success rate of each child in all imitation and JA tasks were scored as the children’s performance. Figure 5b–d present the scores of Imitation, RJA, and IJA in the three developmental assessment sessions (i.e. Baseline, Pre-Test, and Post-Test). Figure 5b–d indicate that all participants showed improvement in Imitation, RJA, and IJA.

The imitation scores were positively Pearson’s correlated with the subjects’ overall RJA scores at the average baseline, i.e. the mean of the Baseline and Pre-Test (Pearson correlation coefficient r = 0.98, p = 0.01), and the Post-Test (r = 0.99, p = 0.01) sessions. In addition, the participant’s Imitation performances were also positively associated with the IJA scores at the average of the Baseline and Pre-Test (r = 0.99, p < 0.001), and the Post-Test (r = 0.99, p < 0.001) sessions. Moreover, exploring the relationship between the RJA overall scores and IJA performances indicated strong positive relationships at the average baseline (r = 0.98, p = 0.01) and the Post-Test (r = 0.98, p = 0.02).

4.2 Questionnaires

The participants’ parents filled in four questionnaires four times in the Baseline, Pre-Test, Post-Test, and Follow-up Test during 22 weeks. The results of the questionnaires are presented in Figure 6a–d.

Figure 6 The Questionnaires Scores of the LF subjects in the Baseline (Week #1), Pre-Test (Week #6), Post-Test (Week #14), and Follow-up Test (Week #22) filled in by the parents: (a) overall GARS scores (indicating autism severity), (b) ASSP overall scores, (c) Autism Checklist overall scores, and (d) PSI-SF scores.

Figure 6

The Questionnaires Scores of the LF subjects in the Baseline (Week #1), Pre-Test (Week #6), Post-Test (Week #14), and Follow-up Test (Week #22) filled in by the parents: (a) overall GARS scores (indicating autism severity), (b) ASSP overall scores, (c) Autism Checklist overall scores, and (d) PSI-SF scores.

4.2.1 GARS

The results of the GARS questionnaire, which indicates the severity of the subject’s autism, are shown in Figure 6a. It should be mentioned that a lower score in this test indicates lower autism severity. This questionnaire covers Stereotyped Behaviors, Communication, Social Interactions, and Developmental Disturbances subscales. As can be seen, slight decreases in autism severity were reported in three out of the four participants from the Pre-Test to Post-Test. However, the amount of change was different for each subject, and only R1 showed noticeable progress.

According to Table 4, the Cohen’s d effect size regarding the overall scores of the GARS for the LF subjects is very small (i.e. 0.19 < 0.2). However, a noticeable point from this questionnaire is that the calculated effect size in the Stereotyped Behaviors subscale is 0.65 (which is close to being large). This result is similar to the findings of ref. [8,16] (e.g. we noticed a large Cohen’s d in this subscale in ref. [8]). In contrast to ref. [8], in which we indicated a large effect size on the Communication subscale for the HF subjects, in this study, the equivalent Cohen’s d was small (Table 4).

It should be noted that the questions regarding the fourth subscale in the GARS (i.e. Developmental Disturbances) are about the first 36 months of the child’s life; therefore, considering the age of the participants in this study, no changes could be seen in this subscale for the subjects.

4.2.2 ASSP

The results of the ASSP questionnaire are presented in Figure 6b. This questionnaire covers three subscales: Social Reciprocity, Social Participation/Avoidance, and Detrimental Social Behaviors. The figure shows that according to the parents’ viewpoint, the social skills of three out of four participants improved from the Pre-Test to Post-Test with the approximate large effect size (i.e. 0.62 > 0.5). For R1, no changes were reported by the parent regarding the ASSP questions. In ref. [8], the calculated Cohen’s d, regarding the overall ASSP scores, was medium (i.e. ∼0.5).

An overview on the ASSP’s subscales indicates that:

  1. (1)

    The Social Reciprocity scores for all of the subjects improved (with a change in mean and standard deviation of 5.0 and 1.14, respectively). The Cohen’s d effect size regarding this subscale is large (>0.8).

  2. (2)

    The Social Participation had a small change (with a change in mean and standard deviation of 1.0 and 0.8, respectively). In addition, a small effect size was calculated for this subscale, which might show that the designed protocol has only a small effect on the Social Participation of children with LF ASD.

  3. (3)

    The Detrimental Social Behaviors was approximately unchanged for the participants. The calculated effect size is near zero for this subscale. Similar to Social Participation, the designed games did not have a measurable change on the detrimental social behaviors of the LF subjects in this study.

Comparing ASSP scores between the Post-Test and Follow-up Test showed us the retention and persistence of the robot-assisted interventions’ effects on the participants.

Imitation performances of the participants were positively associated with the ASSP overall scores at the average baseline, i.e. the mean of the Baseline and Pre-Test scores (Pearson correlation coefficient r = 0.99, p < 0.001) and the mean of the Baseline and the Post-Test scores (r = 0.96, p = 0.04) sessions. The ASSP total scores were also positively Pearson’s correlated with the subjects’ overall RJA scores at the average baseline (r = 0.99, p = 0.01) and marginally correlated with the Post-Test (r = 0.93, p = 0.07) sessions. In addition, exploring the relationship between the ASSP overall scores and IJA performances indicated a strong positive relationship at the average baseline (r = 0.99, p < 0.001) and the Post-Test (r = 0.97, p = 0.02).

4.2.3 Autism checklist

We present the results of the Autism Checklist questionnaire in Figure 6c. This questionnaire includes “Verbal, Language and Communication,” “Socializing,” “Sensory or Cognitive Awareness,” and “Physical and Behavioral Health” subscales. It indicated that the scores of all participants in this questionnaire improved (even if only slightly) in the course of the program. The change in the mean is 5.5 for the subjects of this study. The parents of R1 and R3 reported improvement in their children for more than eight items on this questionnaire. According to Table 4, the Cohen’s d effect size regarding the overall scores of this questionnaire is large (i.e. 0.90 > 0.8). It should be noted that there are only three choices to answer the questions of the Autism Checklist questionnaire, which makes it insensitive to small changes in the symptoms/behaviors of the children.

A close look at the items and subscales of the Autism Checklist questionnaire gives us the following information:

  1. (1)

    In the “Verbal, Language, and Communication” subscale, on average, the LF subjects showed improvement in two items (i.e. mean changes: 1.75). The effect size of this subscale is close to being large (i.e. 0.76 which is ∼0.8).

  2. (2)

    For three out of four participants, the scores of the “Socializing” subscale improved slightly during the intervention sessions (mean changes: ∼1). Regarding the second subscale, the Cohen’s d effect size is small.

  3. (3)

    Based on the parents’ reports, the effect size in the “Sensory or Cognitive Awareness” subscale is also small.

  4. (4)

    The scores of the “Physical and Behavioral Health” subscale had an approximate medium effect size (i.e. ∼0.5).

4.2.4 PSI-SF

In order to investigate the impact of the robot-assisted interventions on the stress level of the participants’ parents, the PSI-SF questionnaire was filled in by the parents four times (i.e. Baseline, Pre-Test, Post-Test, and Follow-up Test) (Figure 6d). It could be seen that in two out of the four participants (i.e. R3 and R4), the parenting stress decreased during the course of the program. According to Table 4, the Cohen’s d effect size regarding this questionnaire is large. It should be noted that in ref. [8], the calculated effect size was medium.

5 Discussion

In this section, the presented results and graphs of the developmental assessments, including the Stambak Rhythm Test, ESCS/imitation, and the questionnaires, are discussed in order to investigate the impact of the proposed robot-assisted interventions on the social and cognitive skills of the participants with LF ASD. The preliminary answers to the research questions of this study are presented. Due to the small number of subjects in this study, the findings of this research should be considered as only an estimation of using this type of technology for the education/rehabilitation of children with LF ASD and cannot necessarily be generalized to all situations for individuals with LF autism.

5.1 Developmental assessments

5.1.1 Stambak rhythm test

According to the Stambak Rhythm Test results, as a preliminary finding of the study’s first research question, we conclude that social robots do have the ability to teach music rhythms to children with LF ASD; however, it has not been proven if robots can teach music notes and phrases to LF subjects. We think that the proposed schedule for the robot-assisted music education of the LF participants was too substantial, and the time duration for each program’s item and the number of sessions should be extended for these children to (hopefully) obtain better results. We would like to add that although the scores are close to zero for R3 and R4 at the Baseline, they do not represent their abilities to perform the tasks in that session; in fact, getting a non-zero score in every educational class needed both cooperation and having the appropriate knowledge or ability in the test. Fortunately, both R3 and R4 cooperated in the Pre- and Post-Tests, which enabled us to have at least an estimation of their learning curves with regard to their perception of rhythm.

As mentioned in the literature, improvement in children’s rhythm recognition (and reproduction) can positively affect their internal body’s rhythm regulation, talking, walking, and even writing [44]. Moreover, the children’s performance in the Stambak Test shows us acceptable levels of instruction’s perception regarding music education.

5.1.2 Assessing the social and cognitive skills of the participants

Improvement in imitation skills of children with autism via robot-assisted and intelligent systems imitation games [16,21,23,25] and non-robotic imitation games [45,46] is also reported in the literature. These findings are in line with the results of this study.

Improvement in the JA skills of children with autism has been reported in different studies based on robotic and intelligent systems [8,16,21,22,23,47]; however, according to the literature, this statement should be taken with caution. In ref. [47], the authors reported that their robot performed as a catalyst in eliciting JA behaviors for their subjects with ASD. They confirmed that robotic technology could be a useful tool for the cognitive rehabilitation of children with autism. The findings of our study are also in line with the findings of Kalas [6] and Kim et al. [7], which show the improvement potential of performing (non-robotic) music-based clinical sessions on JA skills of children with ASD. It is necessary to mention that improvements in JA due to robot therapy still need to be confirmed, and this common hypothesis in the literature needs to be studied with caution [20,48]. In ref. [48], the authors tried to provide a systematic review about the usefulness of social robots in autism rehabilitation. While considering different questions, such as “Do children with ASD improve their performances in JA tasks if experimenters use a robot?” and “Do children turn their attention to the robot?,” they analyzed five papers to investigate the impact of robot-assisted interventions on JA skills, and then indicated that the results reported in the literature were very diverse and exploratory. The same point was mentioned in ref. [20], and the results presented in the literature have been considered proof-of-concept. The authors of ref. [48] concluded that while robots are very strong attractors during the intervention sessions for children with ASD, they can also become distractors from performing the tasks. The main point is that the authors of ref. [48] believed that the best role for social robots in these situations is the “target role,” and human-mediators are encouraged to design the scenario so that it induces/encourages participants to turn their attention toward the robot. Following music notes with the eyes, hand–eye coordination, and gaze-shifting between the robot and musical instruments are among the most important cognitive situations provided for our subjects with autism in this program. These situations provide an appropriate target role for the robot to affect the subject’s JA skills in the intervention sessions. Therefore, the obtained improvement in the performance of the children regarding JA could be due to the combinatorial nature of our robot-assisted music-based program.

Regarding the second research question (Q2), the proposed intervention has the potential to improve imitation and/or JA skills of children with LF autism. With regard to the social skills, we return again to the ASSP questionnaire.

5.2 Questionnaires

The results of the questionnaires filled in by the parents are discussed in the following subsections of the article.

5.2.1 GARS

Regarding the presented results of the Stereotyped Behaviors subscale of the GARS, we believe that the designed interventions, i.e. the combination of the Nima social robot’s attendance and/or the music games, were somehow able to attract the participants’ attention and motivate the subjects to attend the games effectively. In fact, the robot’s attendance and/or music class environment (most likely both together) may have caused the children to temporarily overlook some of their internal problems and stereotyped behaviors. It should be noted that this study concentrated on the effects of the overall robot-assisted music-based program on participants with LF autism, and we are currently unable to determine the ratio of positive effects of the robot’s attendance and the music games separately. Although the goal of designing the games was not to directly affect a decrease in maladaptive behaviors of the children, based on the parents’ report, the echo and stereotyped behaviors of three out of four participants decreased over the course of the program. In ref. [49], Tapus et al. also mentioned that the stereotyped behaviors of their subject with LF ASD decreased during the child–robot interaction.

5.2.2 ASSP

Regarding the ASSP questionnaire, the amounts of the reported improvement in social skills were different for each subject. This result is in line with the findings of ref. [2,7,11,30,33] in both robot-assisted and non-robotic music-based investigations.

Similar to our findings, previous studies [16,22,23] have mentioned that robot-assisted programs, which include imitation and joint-attention games, have improved the social skills of participants with autism. A systematic look at the principles of reciprocal imitation training programs (such as sitting face-to-face with the child, playing with the toys, imitating facial expressions and body gestures, and playing pretended games) could shed light on the potential of imitation games (e.g. music-based imitation tasks) in improving social skills [45,46]. Putting children with ASD in imitation situations could lead to an increase in their social interactions, more participation in activities and games, and improvement in their social responsibility [46]. It should be noted that one of the goals of designing such robot-assisted music-based games was to (hopefully) affect the social skills of children with ASD. Similar to our findings in this study, Feil-Seifer and Matarić [50] successfully conducted a robot-assisted study using their Bandit robot as a catalyst/facilitator for eliciting social behaviors for some participants with autism.

5.2.3 Autism checklist

Regarding the second subscale of the Autism Checklist questionnaire, the Cohen’s d effect size is small. However, we believe that this is because of the three-choice answer format (i.e. never, sometimes, and always) of this questionnaire, which is hardly sensitive to small changes in participants’ behaviors.

According to the third subscale of the Autism Checklist, it seems that the designed games could not seriously affect this subscale for the children with LF ASD during the program.

Although the scores of the fourth subscale of this questionnaire (i.e. “Physical and Behavioral Health” subscale) had an approximate medium effect size (i.e. ∼0.5), it is most probably due to the development of the children as well as skills taught by the parents and ABA therapists outside of the music classes (and not necessarily because of the robot-assisted interventions) in our program. In fact, we do not expect that the proposed intervention would have any noticeable effect on the fourth subscale of the Autism Checklist questionnaire.

5.2.4 PSI-SF

Regarding the PSI-SF results, a possible reason for the decrease in the stress level of the parents in this study (which we believe is because of our intervention sessions) is that the parents of children with LF ASD (especially children who are less than 6 years old) usually believe that their children are not able to learn any structured and complicated tasks, and any slight improvement during our music-based classes refutes such (incorrect) hypotheses in their minds. Moreover, by taking the time and effort to attend to their children in the class, the mothers felt that they were performing their parental duties well, which directly affects their stress levels. It should be noted that the parents were present in the educational environment during the program, and observing their child’s happiness induces a good feeling for the parents. Booth and Jernberg [51] have also reported that the parents are important elements of the child’s treatment/rehabilitation. As mentioned in ref. [8], the calculated effect size was medium, which could be because the mothers of HF children had higher expectations than the LF parents regarding their child’s ability to learn. Regarding the third research question (Q3) of this study, we can conclude that robot-assisted music-based interventions could reduce the stress levels of the parents of children with LF ASD (with a large effect size). It should be noted that the results of the PSI-SF questionnaire directly depend on the current mental state of the parents, and the increase in the stress level for R4’s mother (from Week #1 to Week #6) could be due to other problems in their life and/or lack of familiarity with the questions when filling out the questionnaire for the first time.

5.2.5 Discussion summary

As a summary of the questionnaires’ section, we need to mention that one should not expect large changes in the skills of the participants with LF ASD during the short time interval of the robot-assisted interventions. Therefore, as it is seen, the improvement in the questionnaires was often slight (but nonetheless important for the children/parents). Questionnaire results showed us that the participants’ parents believed some improvement was made in their child’s education and social and cognitive skills over the course of the program.

To wrap up all the findings and present a summary for the second research question, we concluded that robot-assisted music education classes could slightly improve the social and cognitive skills of children with LF ASD, but the amount of improvement regarding the music education for the LF subjects are not comparable to the HF participants’ performances previously performed/presented in ref. [8]. The participants with LF ASD showed different improvements in social and cognitive skills. We would like to add that this might be due to the combinatorial nature of the designed robot-assisted music-based tasks, which affect different skills of the subjects with LF autism.

In summary, a general slight effect and improvement of the participants’ behaviors were indicated after the robot-assisted music intervention. Having no control group in this study was an important limitation of our research and did not allow us to identify which of the improvements might have happened without the proposed interventions. In order to have an initial exploratory comparison of the results of this study with similar research conducted without robots (but with a similar schedule to this work), we can refer to one of our previous studies in which participants with autism took part in a set of similar interventions with tablet-based music games called “Xylotism” [52]. By calculating the Cohen’s d effect sizes, we saw that the effect sizes in that study were at the same level for the ASSP, Autism Checklist, and PSI-SF overall scores. In this study, we reached a much higher effect size in the “Verbal, Language and Communication” subscale of the Autism Checklist (i.e. 0.76 in the robot-assisted vs 0.55 in the tablet-based study) and the “Stereotyped Behaviors” subscale of the GARS (i.e. 0.65 vs 0.21). However, due to reasons such as the nonhomogeneous autism severity and age ranges of the participants of these two studies, we cannot make any strong/conclusive claims from this initial comparison or determine for sure that the improvement and the learning curve reported in a 5-month long intervention would not have happened even without the robot-assisted intervention.

The proposed platform has the potential to provide a rich communicational–educational environment for children with autism. This study has been done in order to explore the potential and preliminary impacts of using robotic technology in active music classes for children with LF ASD, and the positive findings of this preliminary exploratory study show the potential usefulness of modern technologies in the social and cognitive rehabilitation of individuals with ASD.

6 Limitations and future work

We would like to mention that our team tried their best to control the conducted robot-assisted program; however, there is still a possibility that some of the children’s reported improvement items were attributable to other educations/interventions our subjects may have received outside of our program (that were impossible to be fully controlled). Unfortunately, we did not have a control group to determine if the findings of this study were due to our interventions or not. Therefore, as with similar studies [16,26,53], some of the reported children’s improvements could also be due to other educations/treatments received outside our classes (e.g. parents might, even unconsciously, provide further informal opportunities for their children to improve in the specific skills) that were uncontrollable by us. It should be noted that even the children’s maturation during the study might have affected some of the obtained results. Moreover, the small number of participants was the other main limitation of this study. Increasing the number of children as well as the number of conducted sessions could give us more exact information regarding the impact and potential of such robot-assisted sessions in autism treatment. In our study, we would suggest that the reported results should be considered as exploratory findings. Additionally, the quantitative video coding results from the intervention sessions could not be completed during the Covid-19 situation, so only the qualitative descriptions of the children’s performances during the program were presented. In this stage, the effects of the “robot’s attendance” and the “music-based scenario” are not independently detectable in the children’s reported improvements, and the presented results are considered to be based simultaneously on the robot’s attendance and music games. The noticeable amount of unpredictable and maladaptive behaviors of the participants with LF autism sometimes affected the impact of the sessions (which caused some differences between the findings of this study and our previous work [8]). We would like to encourage other scientists to perform such active music-based protocols (even without the robot) to obtain more details about the potential for positive impacts on children with autism. It should be noted that we are still unable to discuss how and if performing the same program without a social robot and only a human mediator would affect (more/less) the conducted sessions for the children with LF autism; therefore, exploring the entire and exact effects of such robot-assisted music-based interventions still needs further complementary studies.

7 Conclusion

In this study, a proposed robot-assisted music-based intervention program was conducted in a rich communicational–educational environment for four children with LF ASD. The protocol included hierarchical tasks with an emphasis on imitation, JA, and turn-taking games. After a 5-month music education program, we found a large effect size in the rhythm reproduction test, which could show that social robots do have the ability to teach music rhythms to children with LF ASD. Unfortunately, due to the deficits in the participants’ cognitive skills alongside the short time interval of the program, no systematic learning of notes and musical phrases occurred by the children in this study. Moreover, we found that the stereotyped behaviors of our LF participants decreased with an approximate large Cohen’s d effect size. In addition, during the developmental assessments, we indicated some improvement in imitation and JA skills of the LF autism participants, which was discussed and compared to the findings of similar papers. The last finding of the article was the decrease in the stress levels of the children’s parents with an approximate large effect size. All in all, we believe that while there is positive potential in using social robots in intervention sessions with children with autism, one should avoid expecting large improvements and consider the reported findings of this study as preliminary exploratory results.

Acknowledgements

Our profound gratitude goes to the “CTAD” and its psychologists for their contributions to the clinical trials with children with autism. We sincerely appreciate Prof. Hamidreza Pouretemad for his consults during the study. This research was funded by the Sharif University of Technology (Grant No. G980517) and the “Cognitive Sciences and Technology Council” (CSTC) of Iran (http://www.cogc.ir/) (Grant No. 95p22). We also appreciate the “Dr AliAkbar Siasi Memorial Grant Award” for the complementary support of the Social and Cognitive Robotics Laboratory. Also, special thanks to Mrs. Shari Holderread (American English Editor) for the final editing of our paper.

    Funding information: This study was funded by the Sharif University of Technology (Grant No. G980517) and the “Cognitive Sciences and Technology Council” (CSTC) of Iran (Grant No. 95p22).

    Conflict of interest: Alireza Taheri and Ali Meghdari have received research grants from the Sharif University of Technology and the “Cognitive Sciences and Technology Council” (CSTC) of Iran, respectively. Azadeh Shariati, Rozita Heidari, Mojtaba Shahab, and Minoo Alemi declared that they have no conflict of interest.

    Ethical approval: Ethical approval for the protocol of this study was provided by the Iran University of Medical Sciences (#IR.IUMS.REC.1395.95301469).

    Informed consent: Informed consent has been obtained from all individuals included in this study.

    Data availability statement: The data that support the findings of this study are available from the Social and Cognitive Robotics Lab., SUT; but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of the Social and Cognitive Robotics Lab., SUT.

References

[1] M. Geretsegger, U. Holck, Ł. Bieleninik, and C. Gold, “Feasibility of a trial on improvisational music therapy for children with autism spectrum disorder,” J. Music. Ther., vol. 53, no. 2, pp. 93–120, 2016. Search in Google Scholar

[2] A. N. Bhat and S. Srinivasan, “A review of “music and movement” therapies for children with autism: embodied interventions for multisystem development,” Front. Integr. Neurosci., vol. 7, p. 22, 2013. Search in Google Scholar

[3] P. Kern and M. Humpal, Early Childhood Music Therapy and Autism Spectrum Disorders: Developing Potential in Young Children and their Families, Jessica Kingsley Publishers, London, UK & Philadelphia, PA, 2012. Search in Google Scholar

[4] H. A. Lim and E. Draper, “The effects of music therapy incorporated with applied behavior analysis verbal behavior approach for children with autism spectrum disorders,” J. Music. Ther., vol. 48, no. 4, pp. 532–550, 2011. Search in Google Scholar

[5] B. A. Corbett, K. Shickman, and E. Ferrer, “Brief report: the effects of Tomatis sound therapy on language in children with autism,” J. Autism Dev. Disord., vol. 38, no. 3, pp. 562–566, 2008. Search in Google Scholar

[6] A. Kalas, “Joint attention responses of children with autism spectrum disorder to simple versus complex music,” J. Music. Ther., vol. 49, no. 4, pp. 430–452, 2012. Search in Google Scholar

[7] J. Kim, T. Wigram, and C. Gold, “The effects of improvisational music therapy on joint attention behaviors in autistic children: a randomized controlled study,” J. Autism Dev. Disord., vol. 38, no. 9, p. 1758, 2008. Search in Google Scholar

[8] A. Taheri, A. Meghdari, M. Alemi, and H. Pouretemad, “Teaching music to children with autism: a social robotics challenge,” Sci. Iran., vol. 26, Special Issue on: Socio-Cognitive Engineering, pp. 40–58, 2019. Search in Google Scholar

[9] M. J. Lanovaz, S. E. Fletcher, and J. T. Rapp, “Identifying stimuli that alter immediate and subsequent levels of vocal stereotypy: A further analysis of functionally matched stimulation,” Behav. Modif., vol. 33, no. 5, pp. 682–704, 2009. Search in Google Scholar

[10] S. Wood, “A study of the effects of music on attending behavior of children with autistic-like syndrome,” Master’s Theses, San Jose State University, 1991. Search in Google Scholar

[11] K. Simpson and D. Keen, “Music interventions for children with autism: narrative review of the literature,” J. Autism Dev. Disord., vol. 41, no. 11, pp. 1507–1514, 2011. Search in Google Scholar

[12] A. B. LaGasse, “Effects of a music therapy group intervention on enhancing social skills in children with autism,” J. Music. Ther., vol. 51, no. 3, pp. 250–275, 2014. Search in Google Scholar

[13] V. Pasiali, “The use of prescriptive therapeutic songs in a home-based environment to promote social skills acquisition by children with autism: Three case studies,” Music. Ther. Perspect., vol. 22, no. 1, pp. 11–20, 2004. Search in Google Scholar

[14] J. Katagiri, “The effect of background music and song texts on the emotional understanding of children with autism,” J. Music. Ther., vol. 46, no. 1, pp. 15–31, 2009. Search in Google Scholar

[15] C. Carnahan, J. Basham, and S. Musti-Rao, “A low-technology strategy for increasing engagement of students with autism and significant learning needs,” Exceptionality, vol. 17, no. 2, pp. 76–87, 2009. Search in Google Scholar

[16] A. Taheri, A. Meghdari, M. Alemi, and H. Pouretemad, “Human–robot interaction in autism treatment: a case study on three pairs of autistic children as twins, siblings, and classmates,” Int. J. Soc. Robot., vol. 10, no. 1, pp. 93–113, 2018. Search in Google Scholar

[17] K. Baraka, F. S. Melo, and M. Veloso, “Interactive robots with model-based ‘autism-like’ behaviors,” Paladyn, J. Behav. Robot., vol. 10, no. 1, pp. 103–116, 2019. Search in Google Scholar

[18] L. J. Wood, A. Zaraki, M. L. Walters, O. Novanda, B. Robins, and K. Dautenhahn, “The iterative development of the humanoid robot Kaspar: An assistive robot for children with autism,” in: International Conference on Social Robotics, Springer, Cham, 2017, pp. 53–63. Search in Google Scholar

[19] A. G. Pour, A. Taheri, M. Alemi, and A. Meghdari, “Human–robot facial expression reciprocal interaction platform: case studies on children with autism,” Int. J. Soc. Robot., vol. 10, no. 2, pp. 179–198, 2018. Search in Google Scholar

[20] P. Chevalier, K. Kompatsiari, F. Ciardo, and A. Wykowska, “Examining joint attention with the use of humanoid robots - A new approach to study fundamental mechanisms of social cognition,” Psychonomic Bull. Rev., vol. 27, pp. 217–236, 2020. Search in Google Scholar

[21] B. Scassellati, H. Admoni, and M. Matarić, “Robots for use in autism research,” Annu. Rev. Biomed. Eng., vol. 14, pp. 275–294, 2012. Search in Google Scholar

[22] J.-J. Cabibihan, H. Javed, M. Ang, and S. M. Aljunied, “Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism,” Int. J. Soc. Robot., vol. 5, no. 4, pp. 593–618, 2013. Search in Google Scholar

[23] L. I. Ismail, T. Verhoeven, J. Dambre, and F. Wyffels, “Leveraging robotics research for children with autism: a review,” Int. J. Soc. Robot., vol. 11, no. 3, pp. 389–410, 2019. Search in Google Scholar

[24] J. J. Diehl, L. M. Schmitt, M. Villano, and C. R. Crowell, “The clinical use of robots for individuals with autism spectrum disorders: A critical review,” Res. Autism Spectr. Disord., vol. 6, no. 1, pp. 249–262, 2012. Search in Google Scholar

[25] A. Duquette, F. Michaud, and H. Mercier, “Exploring the use of a mobile robot as an imitation agent with children with low-functioning autism,” Auton. Robot., vol. 24, no. 2, pp. 147–157, 2008. Search in Google Scholar

[26] A. Taheri, A. Meghdari, M. Alemi, and H. Pouretemad, “Clinical interventions of social humanoid robots in the treatment of a set of high-and low-functioning autistic Iranian twins,” Sci. Iran., vol. 25, no. 3, pp. 1197–1214, 2018. Search in Google Scholar

[27] Z. Telisheva, A. Turarova, A. Zhanatkyzy, G. Abylkasymova, and A. Sandygulova, “Robot-assisted therapy for the severe form of autism: challenges and recommendations,” in: International Conference on Social Robotics, Springer, Cham, 2019, pp. 474–483. Search in Google Scholar

[28] L. Boccanfuso, S. Scarborough, R. K. Abramson, A. V. Hall, H. H. Wright, and J. M. O’Kane, “A low-cost socially assistive robot and robot-assisted intervention for children with autism spectrum disorder: field trials and lessons learned,” Auton. Robot., vol. 41, no. 3, pp. 637–655, 2017. Search in Google Scholar

[29] T. Zorcec, B. Robins, and K. Dautenhahn, “Getting engaged: assisted play with a humanoid robot Kaspar for children with severe autism,” in: International Conference on Telecommunications, Springer, Cham, 2018, pp. 198–207. Search in Google Scholar

[30] Gifford T., S. Srinivasan, M. Kaur, D. Dotov, C. Wanamaker, G. Dressler, et al., Using robots to teach musical rhythms to typically developing children and children with autism University of Connecticut, pp. 1–6, 2011, 11. Search in Google Scholar

[31] W. C. So, M. K. Y. Wong, W. Y. Lam, C. H. Cheng, S. Y. Ku, K. Y. Lam, et al., “Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition,” Res. Dev. Disabil., vol. 86, pp. 62–75, 2019. Search in Google Scholar

[32] Y.-H. Peng, C.-W. Lin, N. M. Mayer, and M.-L. Wang, “Using a humanoid robot for music therapy with autistic children,” in: 2014 CACS International Automatic Control Conference (CACS 2014), IEEE, Kaohsiung, Taiwan, 2014, pp. 156–160. Search in Google Scholar

[33] S. M. Srinivasan, I.-M. Eigsti, T. Gifford, and A. N. Bhat, “The effects of embodied rhythm and robotic interventions on the spontaneous and responsive verbal communication skills of children with Autism Spectrum Disorder (ASD): A further outcome of a pilot randomized controlled trial,” Res. Autism Spectr. Disord., vol. 27, pp. 73–87, 2016. Search in Google Scholar

[34] J. Y. Lin, Y. H. Ma, S. Cosentino, and A. Takanishi, “Development and evaluation of a quantitative analysis system for robot-assisted musical therapy,” Delphi, vol. 2, no. 1, pp. 35–42, 2019. Search in Google Scholar

[35] B. Bankson, A. Cox, H. Fulmer, L. Hausman, and D. Longfellow, “Outcomes of music, dance, & movement interventions for children with autism spectrum disorder within the scope of occupational therapy: scoping review,” Presented in Partial Fulfillment of the Master of Health Sciences in Occupational Therapy degree at Augusta University, Georgia, United States, 2020. Search in Google Scholar

[36] P. Mundy, C. Delgado, J. Block, M. Venezia, A. Hogan, and J. Seibert, “Early social communication scales (ESCS),” A Manual for the Abridged, University of Miami, Coral Gables, FL, pp. 1–65, 2003. Search in Google Scholar

[37] H. Pouretemad, Diagnosis and Treatment of Joint Attention in Autistic Children, Tehran, Iran: Arjmand Book, 2011. Search in Google Scholar

[38] H. Gardner, “Children’s duplication of rhythmic patterns,” J. Res. Music. Educ., vol. 19, no. 3, pp. 355–360, 1971. Search in Google Scholar

[39] S. Bellini and J. K. Peters, “Social skills training for youth with autism spectrum disorders,” Child. Adolesc. Psychiatr. Clin. North. Am., vol. 17, no. 4, 857–3873, 2008. Search in Google Scholar

[40] J. E. Gilliam, Gilliam Autism Rating Scale: Examiner’s Manual, Austin, TX: Pro-Ed, USA, 1995. Search in Google Scholar

[41] R. R. Abidin, Parenting Stress Index (PSI), Charlottesville, VA: Pediatric Psychology Press, 1990. Search in Google Scholar

[42] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, Academic Press, Lawrence Erlbaum Associates, Inc., USA, 1977. Search in Google Scholar

[43] B. Scassellati, L. Boccanfuso, C. M. Huang, M. Mademtzi, M. Qin, N. Salomons, et al., “Improving social skills in children with ASD using a long-term, in-home social robot,” Sci. Robot., vol. 3, no. 21, p. 7544, 2018. Search in Google Scholar

[44] D. A. Cordell, “Peters, J. S. (2000). Music therapy: An introduction (2nd ed.). Springfield, IL: Charles C. Thomas Publisher. 474 pages. ISBN 0-398-07042-3. $79.95 cloth. $64.95 paper,” Music. Ther. Perspect., vol. 20, no. 1, p. 48, 2002. Search in Google Scholar

[45] B. Ingersoll and S. Gergans, “The effect of a parent-implemented imitation intervention on spontaneous imitation skills in young children with autism,” Res. Dev Disabil. vol. 28, no. 2, pp. 163–175, 2007. Search in Google Scholar

[46] P. Moghim-Islam, M. Pourmohammadreza-Tajrishi, and H. Haghgou, “The Effect of reciprocal imitation training on social skills of children with autism,” J. Rehabil, Spec. Issue: Pediatr. Neurorehabil., vol. 14, pp. 59–67, 2014. Search in Google Scholar

[47] B. Robins, P. Dickerson, P. Stribling, and K. Dautenhahn, “Robot-mediated joint attention in children with autism: A case study in robot-human interaction,” Interact. Stud., vol. 5, no. 2, pp. 161–198, 2004. Search in Google Scholar

[48] P. Pennisi, A. Tonacci, G. Tartarisco, L. Billeci, L. Ruta, S. Gangemi, S., et al., “Autism and social robotics: A systematic review,” Autism Res., vol. 9, no. 2, pp. 165–183, 2016. Search in Google Scholar

[49] A. Tapus, A. Peca, A. Aly, C. Pop, L. Jisa, S. Pintea, et al., “Children with autism social engagement in interaction with Nao, an imitative robot: A series of single case experiments,” Interact. Stud., vol. 13, no. 3, pp. 315–347, 2012. Search in Google Scholar

[50] D. Feil-Seifer and M. J. Matarić, “Toward socially assistive robotics for augmenting interventions for children with autism spectrum disorders,” in: Experimental Robotics, Springer, Berlin, Heidelberg, 2009, pp. 201–210. Search in Google Scholar

[51] P. B. Booth and A. M. Jernberg, Theraplay: Helping Parents and Children Build Better Relationships Through Attachment-Based Play, John Wiley & Sons, Printed in the United States of America, 2009. Search in Google Scholar

[52] M. T. Elahi, A. H. Korayem, A. Shariati, A. Meghdari, M. Alemi, E. Ahmadi, et al., ““Xylotism”: A tablet-based application to teach music to children with autism,” in International Conference on Social Robotics, Springer, Cham, 2017, pp. 728–738. Search in Google Scholar

[53] A. Taheri, A. Meghdari, and M. H. Mahoor, “A close look at the imitation performance of children with autism and typically developing children using a robotic system,” Int. J. Soc. Robot., 2020, https://doi.org/10.1007/s12369-020-00704-2. Search in Google Scholar

Received: 2020-04-01
Revised: 2021-03-16
Accepted: 2021-03-31
Published Online: 2021-04-28

© 2021 Alireza Taheri et al., published by De Gruyter

This work is licensed under the Creative Commons Attribution 4.0 International License.