Impacts of using a social robot to teach music to children with low-functioning autism

1 Introduction

Music lessons are an effective tool to engage children with autism spectrum disorder (ASD) in non-verbal and rhythmic communications [1]. It is reported in ref. [2] that at least 12% of worldwide interventions for individuals with autism have included music-based games in the rehabilitation/treatment process. Ample evidence which demonstrates that playing or even teaching music in intervention sessions could significantly increase the impact of the rehabilitation process for children with autism [3,4]. In ref. [3], the authors emphasized that early behavioral clinical programs are very important for children with ASD, and music therapy can be an effective treatment program. In ref. [4], the authors studied how the perception of musical patterns incorporated in applied behavior analysis verbal behavior (ABA VB) operants impacted the production of speech in 22 (3 to 5-year-old) participants with autism. They reported that both speech and music exercises were beneficial for the production of the ABA verbal operants, and they concluded that music incorporated ABA VB training was most effective in echoic production. Certain previous studies have inspired other scientists to facilitate the multi-system development of individuals with special needs by using an embodied music-based method/approach [2]. Music-based tasks involved in these intervention sessions included playing pre-recorded files and/or teachers/therapists playing musical instruments [2,5]. Kalas [6] and Kim et al. [7] reported that the eye-contact, turn taking, and Joint Attention (JA) skills of the participants with ASD improved through their active music-based sessions. References [8–10] indicate that conducting music-based intervention sessions led to a decrease in self-injuries and stereotyped behaviors in their subjects with autism. In addition, it has been shown that music-based intervention sessions can improve gestural and verbal communication [11], social and emotional skills [12,13,14], and behaviors [15] of children with autism. According to ref. [2], a serious gap in this field of study is the lack of investigations on the impact of music-based treatment protocols on gross/fine motor skills of individuals with autism.

The use of social robots in the education and cognitive rehabilitation of children with autism has been widely investigated during the last two decades [16,17,18,19,20]. Some review papers have discussed, summarized, and argued [21,22,23,24] the findings of different studies in this field of research. The study in ref. [21] mentions that engagement and attention levels of individuals with autism seem to improve during robot-assisted intervention sessions as well as novel social behaviors (such as spontaneous imitation and JA) elicited from participants with ASD. The mentioned article presents an overview of the physical appearance, complexity, and degrees of freedom of robots used in autism therapy. In this regard, the authors concluded that there is no clear-cut agreement on the preferred shape of robots in this area and that humanoid-robots, animal-like robots, and even cartoon-based robots ones have shown a variety of positive impacts on children with ASD in intervention sessions. Moreover, in ref. [21], readers can find comparisons of different human–robot interactions (HRI) and various roles of robots in the field of autism that are focused on encouraging imitative behaviors, eliciting JA, and mediating turn-taking behaviors between the participants and human therapists. All in all, ref. [21] presents an overall positive view of using robots as potential tools in autism rehabilitation. Cabibihan et al. [22] separated the role of robots in autism interventions into friendly playmates, agents for eliciting behaviors, social mediators, social actors, and/or personal therapists. They mentioned that robots are less complex and intimidating than humans for participants with ASD, which made robot-assisted sessions more effective for them. The authors emphasized that future experiments should be designed across different research groups. Another article [23] reviewed more than 400 papers analyzing the use of social robots in autism therapy/rehabilitation. Along with the promising results reported for robot-assisted interventions for children with autism, the authors of ref. [23] emphasized three serious gaps in this field including diversity in the focus of the studies, contribution towards autism impairments, and effectiveness of the interventions. The authors in ref. [24] present a critical review of robot-assisted interventions for children with ASD considering users’ responses to robots as well as the use of robots in eliciting behaviors, modeling skills, and providing appropriate feedback. They indicated that most of the reported results were exploratory with serious limitations, which means they do not necessarily provide strong evidence for generalization or advocate the broad use of robots in autism clinics. So far, most robot-assisted interventions have been performed/published for children with high-functioning (HF) autism [21,23], and unfortunately, the study and comparison of such technological treatment protocols for children with low-functioning (LF) autism has received much less attention [25,26]. It should be noted that on the autism spectrum, individuals with autism who are or appear to be closer to normal/typically developing people are considered to be HF. In general, the ability to communicate verbally and the amount of verbal speech, and/or the awareness of social conventions for individuals with HF autism are higher than peers with LF autism [26,27], while the deficits of children with LF autism are often more visually and aurally obvious to the casual observer.^[1] Working with children with LF autism is very challenging; they may show aggression and other difficult behaviors (due to the combined effects of linguistic/cognitive impairments) [27]. However, the mentioned distinctions are not absolute, and individuals with autism have individual strengths and needs and may behave completely different in various situations [8,26,27].

Considering the mentioned potential of (1) music education/therapy and (2) robot-assisted interventions for children with ASD, in this study we have combined these two paradigms (again) to present an educational-therapeutic environment that can provide a variety of educational and socio-cognitive situations/tasks for participants with autism. To this end, this article endeavors to present the results of running robot-assisted music-based intervention sessions with an emphasis on children with LF ASD. Therefore, an attempt is made to investigate the impacts of teaching music to four LF subjects via a drum/xylophone player NAO robot with regard to (1) music education, (2) social and cognitive skills, and (3) parenting stress levels of the participants during a case study. The current research is built on our previous studies using a social robot to systematically teach music to children with autism [8]. While the focus of our previous study was on children with HF autism, in this article, all of the recruited participants have been diagnosed with LF ASD, which will most likely make the educational interventions more challenging for the teachers. It should be noted that children with ASD, especially LF children, have deficits that usually do not allow complex brain connectivity and flexibility. This motived us to study children with LF autism to ascertain the following: (1) a more in-depth understanding of the unknowns and potentials of LF children’s behaviors during HRI since almost all of the previous works on LF participants with robots have been exploratory [26,27,28,29,30], (2) a better understanding of LF impairments and the differences on the autism spectrum to provide the necessary information to design/propose appropriate robot-assisted intervention protocols, and (3) to study whether the positive changes happening/reported for the LF participant (and/or HF subjects) in ref. [8] could be generalized to other children with LF autism. As previously mentioned, there is ample evidence that this type of intervention is successful when applied by a human therapist; however, it is necessary to note that while the robot plays as an attractive assistive tool for human-therapists to make the education/rehabilitation process more appealing in such studies, in our experiments any unpredictable situations during the sessions were handled by the human teacher. In other words, as the study protocol is described throughout this manuscript, we prefer to have both (1) the knowledge, experience, and influence of the teacher, as well as (2) the social robot’s potential in any education/therapeutic process for music activities (Robots have been shown to be able to facilitate children’s learning process from different aspects [such as music education/therapy area], by providing interactive, spontaneous, consistent, and repetitive trainings for users [8,21,29,30]). In ref. [31], the authors tried to find a preliminary answer to the following question: “Who is a better teacher for children with low-functioning autism, human-mediators or robots?” After conducting their study with 23 LF autism participants, they reported that when the provided tasks are highly structured, it does not matter who (i.e. human or robot) serves as the teacher for the subjects.

Conducting robot-assisted intervention sessions to systematically teach music to children with ASD has some rehabilitation potential. However, to the best of our knowledge, the impact of using social robots in such classes (especially robots that play musical instruments) on the educational and cognitive skills of participants with autism is covered less in the literature [8,30,32,33,34,35]. In ref. [30], a NAO robot was programmed to play a drum to provide consistent repetitive training for rhythm education for participants with ASD and Typically Developing subjects. The main findings of this work confirmed the potential of such studies in improvement in JA skills, conversation bouts, and interlimb/intralimb coordination of the subjects. In another study [32], a xylophone-player Darwin robot was used to conduct a set of imitation tasks during a 7-week program for two children with ASD. While the authors did not observe any significant progress in the participants’ imitation skills, they did indicate some improvement in the communication and concentration of the children with autism. In ref. [33], the authors investigated the importance of robot-assisted music-based sessions on the social attention patterns of 36 participants with autism by conducting a set of scenarios (e.g. rhythm, robotic, and comparison interventions). The role of the robot was to involve the children in signing, making requests, and some cognitive tasks. It should be noted that no musical instrument was played by their NAO robot. In ref. [35], the authors reviewed 13 papers (including both robot- and non-robot-assisted studies) that addressed music, dance, and movement-based interventions for children with autism. One of the studied themes reported that music with instruments can improve fine motor skills, social skills, and bilateral hand coordination in participants with ASD. In ref. [8], we proposed a novel hierarchical robot-assisted music-based protocol to teach the fundamentals of music and playing a drum/xylophone to individuals with ASD. After conducting the interventions for three children with HF, and one child with LF autism, we found interesting progress regarding the rhythm and music notes education, as well as some improvements in social and communication skills of the HF subjects. Our previous study inspired us to conduct the current research. Due to the obvious difference in the learning rate and performance of the LF subject in moderate/high level music tasks compared to the participants with HF autism in ref. [8], we estimated that the designed robot-assisted music-based schedule was too cumbersome for the participant with LF autism. Therefore, we proposed that one possible way to make such classes more effective for children with LF ASD would be to slow down the teaching process (as in the current study) and/or increase the number or density of the interventions.

2 Methodology

2.3 Experimental setup and intervention protocol

This study was conducted at the Social and Cognitive Robotics Lab at Sharif University of Technology, Iran in the presence of the child, the Nima robot, one human teacher, and a robot operator in a Wizard of Oz style control. The room size was approximately 3.5 × 4.5 × 3 m³. The schematics of the intervention room is presented in Figure 1. While it is more common to separate the examination and observation areas in such studies, we unfortunately did not have access to a better/bigger room to conduct our intervention sessions. Nonetheless, based on our previous experience in working with children with autism during robot-assisted interventions, we realized that children less than 8 years-old (especially those with noticeable cognitive deficits such as LF subjects) do not usually understand that the robot is operated by a human operator and do not pay (even moderate) attention to the operator during the sessions. Based on our observations during this study, we did not note any participant distractions because of the operator’s attendance during the sessions. The program included one session of Baseline (Week #1), a Pre-Test (Week #6 before starting the first music intervention session), nine robot-assisted music-based educational sessions (Weeks #6–14), a Post-Test (Week #14 after the last music session), and a Follow-up Test (Week #22, 2 months after the last session). Each intervention session took about 20–30 min.

Figure 1

The schematic of the intervention room and its main components.

During the sessions, the participants attempted to perform the robot/teacher’s instructions in a set of imitation, JA, and turn-taking music-based games. The designed music games/tasks are presented in Table 2. All of the pre-set/structured music activities contained systematic and hierarchical music tasks, including an orientation/introduction session, striking the drum/xylophone bars using the mallets, matching the played colors, hitting the bars with and without counting, rhythm perception, working memory games, learning notes, and playing musical phrases. In the virtual xylophone game (i.e. Game #10), each child stands in front of the Kinect sensor and observes eight colored music bars (corresponding to our xylophone) on the screen. The subject should hit the appropriate sequence of bars per the teacher/robot’s instructions with a colored object or mallet (e.g. in this study, we gave a green hat as the mallet to the children and asked them to play the determined notes). The protocol prompt design is shown in Figure 2. Considering that the LF children’s behaviors were often unpredictable during the sessions (e.g. wandering around the game room, having maladaptive behaviors, hitting Nima, inattention to the robot/teacher, etc.), all of the robot’s commands (as well as the selected exercises with the appropriate difficulty levels) were sent to Nima by the robot operator during the sessions. It should be noted that the criteria for passing an exercise and moving on to the next harder task was to correctly play that exercise for at least 50% of the performed time.

Table 2

The music-based robot-assisted interventions schedule

Hierarchical tasks	Outline	Short description of task modes		Goals and potential impacts
1	Orientation session	a	Introduction of the robot to the participants	– Familiarize the subjects to the class environment and the Nima robot
			Robot’s performance includes greeting, singing, dancing, etc.
2	Strike musical instruments using mallets	a	No bars are removed from the instrument (Easy mode)	– Learn how to make sounds with the xylophone and drum – Imitation – Visual pursuit – Attention
			After each robot’s strike, participant should hit any arbitrary bar of the xylophone
		b	Some of the bars are removed from the xylophone (Difficult mode). Repeat 2a with fewer bars on the instrument
		c	Repeat 2a while the participant uses both of his/her hands in order to play the xylophone and drum
		d	Repeat 2b while the participant uses both of his/her hands in order to play the xylophone and drum
3	Match colors during the music tasks	a	Considering bars’ colors, the participant should imitate Nima’s strike	– Color recognition – Imitation – Instruction perception – Joint attention
		b	Nima strikes a xylophone bar and the participant should point to the rectangle with the same color on the screen (and vice versa)
		c	Participants should find and hit the same color on the instrument as Nima’s eyes’ LED color
4	Hit the instruments and counting	a	Participants’ strikes imitate those of the Nima robot considering the number/order of the played notes. The participant and Nima count “one-two” while hitting the instruments	– Perception of the numbers and counting – Verbal short-term memory
		b	Repeat 4a while saying “Baam–Baam” instead of “one-two”
5	Hit the bars without counting	a	The subject should imitate the notes Nima played without counting	– Imitation – Auditory discrimination – Complex tasks’ instructions
6	Rhythm perception	a	Nima strikes 3 to 5 notes on the instruments in order with different rhythms. The participant should imitate the robot considering the order and the time interval between the played notes	– Auditory imitation skill – Visual pursuit – Hand –eye coordination
7	Working memory game	a	The participant should remember the colors and play them on the xylophone as shown/said by Nima	– Working memory – Joint attention – Dual task performance
8	Teach/read the music notes	a	Nima teaches the music notes (i.e. Do, Re, Me, etc.) and then shows some cards to the participant. The subject should say the name of the notes and strike the equivalent bar on the instrument	– Teach the music notes – Shifting attention skill – Auditory memory
9	Playing music phrases	a	Participant plays short music phrases while saying the notes’ names. Nima assesses the subject’s performance, i.e. playing preliminary/easy songs. Parents can also attend the session and see their child’s performance	– Cooperation between behavior and motor skills – Auditory memory – Play music phrases
10	Playing the virtual xylophone and TURN-taking games	a	Nima hits certain bars. The subject should imitate the same notes on the virtual xylophone (and vice versa)	– Sharing attention between different tasks – Turn-taking – Involve the child in Child–Robot –Teacher games – Generalization skills
		b	The teacher or parent hits the real/virtual bars. The participant and the robot imitate the same notes on the virtual and real xylophone, respectively. (It is expected that the subject generalizes the learnt music skills on the virtual xylophone)

Figure 2

The structure of the triadic Robot–Child–Teacher in the program.

It should be mentioned that because of the LF children’s deficits in instruction perception and cognitive skills, the “learning notes” and “playing musical phrases” items (i.e. Games #8 and #9) could not be systematically performed for our subjects, and we jumped to Game #10 (i.e. playing the virtual xylophone) in the last session. The LF childrens’ imitation of the robot performing the tasks during the educational sessions was also significantly weaker than the HF participants in ref. [8] in all of the schedule items (which is to be expected), even in the striking two note tasks. It should be noted that (correctly) imitating the Nima robot striking three or more notes in order (while considering the time interval between each note) was often hard/challenging even for the HF subjects, and the participants of this study were rarely able to pass such items (e.g. with appropriate counting). Figure 3 shows snapshots of playing the real and virtual xylophones in the intervention sessions.

Figure 3

Snapshots of three of the participants playing the real and virtual xylophones in the intervention sessions: (a) considering bars’ colors, the child should imitate Nima’s strikes (i.e. Game #3-a). (b) The child is imitating the robot’s strikes on the virtual xylophone (i.e. Game #10-a). (c) The child did not do the tasks appropriately based on the instructions of playing the virtual xylophone (i.e. Game #10-a).

2.4 Assessment tools

Two kinds of assessments were conducted to find preliminary exploratory answers to the research questions of this study (Figure 4). Firstly, the Developmental Assessments, which included assessing (1) the imitation and JA skills of the children by extracting and performing 30 tasks from the Early Social Communication Scales (ESCS) [36,37] and regular imitation exercises in autism clinics, as well as (2) assessing the rhythm reproduction ability of the subjects by a human therapist performing the Stambak Rhythmic Structures Reproduction Test [38] three times in the Baseline, Pre-Test, and Post-Test. Regarding the imitation assessments, the subject should perform approximately 15 gross and fine motor imitation exercises presented by the teacher, including the gross/fine imitation of whole body movements and several imitation activities with toys and objects (such as the opposite arm–leg task, balloon game, arranging the objects, match the ribbons, the car game, matches game, etc.; Figure 4a). The percentage of successful performances is considered as the imitation score of each participant. To assess JA, different objects/tasks were presented to the children based on the ESCS instruction to obtain the number of eliciting responses (i.e. Responding to Joint Attention [RJA]) and initiations (both automatic initiation and per the teacher’s requests) to the provided situations (i.e. Initiating Joint Attention [IJA]) with the human-mediator. Again, the percentages of the successful performances for each subject are considered as the RJA and IJA scores, respectively.

Figure 4

Snapshots of the assessment sessions: (a) ESCS/imitation test and (b) Stambak test.

Secondly, four Questionnaires, including the Autism Social Skills Profile (ASSP) [39], Gilliam Autism Rating Scale (GARS) [40], Autism Checklist [5], and Parenting Stress Index-Short Form (PSI-SF) [41], were filled in by the parents four times during the Baseline, Pre-Test, Post-Test, and Follow-up Test sessions. The assessment sessions (i.e. the Baseline, Pre-Test, Post-Test, and Follow-up Test) were conducted without the presence of the Nima robot.

In this study, the Cohen’s d effect size (which is independent of the study’s sample size) [42] between the Pre- and Post-Tests on the questionnaires is reported in Section 4. The Cohen’s d effect size may not represent an actual improvement in the children’s abilities, and it is not presented to make a strong claim or generalize the findings in this study. However, based on the limited available data (which makes it impossible to conduct statistically significant tests), the Cohen’s d effect size can be used to reveal the potential of the designed music-based program’s effects on the participants with LF autism. Moreover, taking inspiration from ref. [43], we also calculated and reported the Pearson correlation coefficients for the cases where a significant association existed between the developmental assessments and questionnaires’ results in Section 4.

3 Qualitative observations of participants’ performances during interventions

After performing the qualitative video analysis of the sessions, the important descriptive observations and recorded progress trends of our four participants are reported in this section. Moreover, a review of the results for the LF subject in ref. [8] is also presented for an initial comparison with the provided qualitative analysis of the sessions of this study.

R1: She had low visual attention in Sessions 1–3. She knew the colors and recognized the robot’s eye color correctly. She interacted with Nima under the guidance of the teacher in the first three sessions. One-hit imitation tasks were performed correctly by the child on both the xylophone and the drum. The sequence of using her hands gradually improved. When playing the instruments, it was easier for her to start her hits with her right hand. R1 did not do badly in imitating the rhythm on the drums when she paid enough attention to the tasks. Imitation of simultaneous strikes by two mallets on two xylophone bars spaced more than one note apart was commonly mistaken. R1 was not very confident in performing two-hit tasks on the xylophone. Usually, the duration of the child’s attention was short during the sessions. In Sessions 4–6, the human teacher reported that her eye contact increased in comparison to Sessions 1–3. The child was helped by the human-mediator in performing two-hit tasks with both hands in order. She imitated the rhythms on the drum under the direct guidance of the teacher and understood the counting technique better than saying “Baam–Baam” when performing the tasks. The child’s attention and cooperation somewhat increased in Sessions 4–6. She voluntarily communicated with the robot and performed the robot’s exercises without much interference from the teacher. She did not perform well in the tasks that she had to start with her left hand. In 4-hit tasks on the drum, she could only perform the task via counting and only with one of her hands (preferably the right hand). In Sessions 7–9, it was easier for her to communicate with Nima than before. The number of consecutive exercises she participated in was short and the child was often distracted from the music tasks. A short musical phrase in the form of some consecutive colors was given to the child; she performed very well in recognizing the order of the notes on the xylophone. However, her performance was not good in terms of quality and consistency of playing the notes. On the virtual xylophone, she realized the connection between the hat and the image. R1 did the exercises very well with appropriate speed (Figure 2b). In the end, it seems that the child has acquired the basic skills needed to learn music and can gradually imitate simple melodic sentences on the xylophone. One of the most important observations was the impact of the sessions on the child in terms of increased interaction and communication (with the teacher and the robot) in contrast to the first two sessions when she avoided even looking at the robot.

R2: The child did not cooperate in Sessions 1 and 2 and did not follow the teacher’s instructions. He was interested in the task in which the robot eye color changed but did not cooperate in other exercises. On occasions, he refused to cooperate with any of the methods of encouragement, and it was very difficult for the teacher to control him. In Sessions 4–6, R2 did some of the exercises presented by the robot. The teacher understood that when the child performed the movements in the standing position, his cooperation increased slightly. This child paid attention and was attracted to each exercise for about five minutes; after that, R2 got bored, and the exercise had to be changed. He also performed worse in some of the intermediate sessions (e.g. Sessions 4–6) than in his first performances. Essentially, he showed resistance to all educational classes. R2 repeated repetitive and meaningless words (related to movies he had seen) to himself during the exercises. In Sessions 7–9, his cooperation increased a bit, and he played some notes on the xylophone, but he still showed a lot of resistance and was not compliant. The incentives that seemed to be useful for others did not affect his performance. He liked the virtual xylophone very much and realized the related exercises; however, he did not cooperate and did not do the tasks based on the instructions. Instead of playing the notes via the colored hat, he put it on and just looked at the virtual bars on the screen (Figure 2c). At the conclusion of the interventions, R2 was only able to perform the one-hit and two-hit exercises on the xylophone and drum to a good extent and hit the relevant note with the guidance of the robot. Overall, the participant was unable to show significant progress during these sessions due to his low functioning autism and consistent lack of cooperation.

R3: In Sessions 1–3, he was very excited and ran around the room. It was very difficult to control the child’s movements. The robot distracted the child in the first two sessions. R3 had no control over the mallets’ hits on the xylophone and hit randomly. He even showed maladaptive behaviors by hitting the mallets on the body of the instrument. With the guidance of the teacher and food incentive, he could only correctly imitate a few one-hit imitation movements on the xylophone when most of the bars were removed from the xylophone. R3 did not know all of the colors (such as purple). In Sessions 4–6, he began to hit the correct bars and no longer hit the body of the instrument. R3 could understand the exercises. In hitting only one bar, he could tell where Nima had hit the xylophone when all of the bars existed on the instrument. The order of using hands for hitting the drum improved compared to the first sessions. Incentives had a significant impact on his performance; however, they made it more difficult to control the child’s behavior. Having stereotyped behavior, he sometimes laughed a lot for no reason. Mirror imitation with the guidance of the human mediator was relatively good. The one-hit imitation on the xylophone was great. R3 could only perform the two-hit imitation with the help of the teacher and a very small number of bars. He usually did it by the trial and error method. In Sessions 4–6, while he was not yet able to identify the bars, he could detect the range positions of the bars played by the robot. He had become curious about the robot. In Sessions 7–9, he liked the virtual xylophone very much. R3 realized the instruction of the virtual game very quickly and used the colored hat to play the virtual instrument well. He recognized the spatial direction of the hand movement well. Sometimes he showed maladaptive behaviors by throwing the hat around. Verbal–motor coordination obviously improved for the child. It was great when he performed two hits correctly without the guidance and control of the human mediator. R3 was able to perform up to three hits on the xylophone and four hits on the drum with the help of counting, although he sometimes made mistakes in the order of the movements. He could control the power when hitting with the mallets. His cooperation increased when he reacted appropriately to the robot’s greetings. He was occasionally not patient enough to wait until the performance of the robot was complete.

R4: In Sessions 1–3, the subject was easily distracted. R4 did not have the desire to interact with the robot himself; but, he interacted with the robot under the guidance of the teacher. He was able to recognize the robot’s eye color properly and played the matching notes on the xylophone. At the beginning (i.e. Sessions 1 and 2), R4 did not use the correct hand to play the music; however, after a little practice, he was able to use both his right and left hands appropriately. He was able to hit the single note on the xylophone correctly with the help of the teacher; however, he had serious difficulties in the two-hit exercises. By performing some movements on the child’s body, the human mediator gradually guided him and improved his performance in playing the notes. After a few sessions, he imitated two-hit tasks with a single mallet correctly. For this subject, persuasive encounters worked better than verbal incentives. In Sessions 4–6, he had a better relationship with Nima and the teacher’s involvement gradually reduced. R4 understood the exercises correctly and performed his tasks independently, and his concentration increased when performing the tasks. He implemented mirror imitation little by little with the help of the teacher and he was able to then perform these tasks independently. The child’s verbal–motor coordination in performing the exercises was still weak at this level. However, he imitated two-hit tasks correctly. In Sessions 7–9, he understood how the virtual xylophone worked during a complex task. However, he could not initially figure out how to use the hat to play the notes. The human mediator believed that verbal–motor coordination at the end of the program was relatively at the proper level for R4. He played up to 3-hit tasks correctly and held the mallets appropriately. R4 had good cooperation for playing the virtual xylophone, and he was able to independently play a short melody on the virtual xylophone. Overall, the child’s performance in Sessions 7–9 seemed to be significantly improved.

Table 3 summarizes the main difficulties and improvements of the subjects during the program. In summary of this section, in both this study and ref. [8], we indicated that the proposed intervention was too cumbersome for the LF subjects. Moreover, expecting participants with LF autism to use both hands consecutively to play the xylophone and the drum as well as learning the notes systematically (not observed in any of our subjects) seems to be a set of very difficult goals, especially in less than 10 sessions. To learn 3-hit tasks on the xylophone using both hands in order and 4-hit tasks on the drum while counting seems to be more logical expectations/goals for these children in similar music-based programs.

4 Results

In this section, the results of the developmental assessments, including the Stambak Rhythm Test, the ESCS/Imitation, and the questionnaires, are presented. The discussion of the results and their comparison to other works are presented in the next section. We also present a comparison of this study’s findings with our previous paper, conducted with children with HF autism [8], considering the interventions’ effect size (Table 4).

4.1 Developmental assessments

The results of the developmental assessment tests in the Baseline, Pre-Test, and Post-Test are shown in Figure 5a–d.

Figure 5

The results of the developmental assessments sessions for the LF subjects in the Baseline (Week #1), Pre-Test (Week #6), and the Post-Test (Week #14): (a) Stambak Rhythm Test, (b) imitation success percentage scores, (c) RJA scores, and (d) IJA scores.

4.1.1 Stambak rhythm test

Figure 5a shows the results of the Stambak Rhythm Test in the Baseline, Pre-Test, and Post-Test. This test contains 21 easy to hard level musical rhythmic actions that were performed on the drum to assess the children regarding their music education.

Figure 5a shows that all of the LF participants showed some improvement, even though slight, in rhythm recognition/reproduction from the Pre-Test to Post-Test, which shows that some educational process occurred regarding the rhythm reproduction. However, none of the LF subjects achieved the musical notes phrase item, nor the reading music phrase item in this study. The calculated Cohen’s d effect size regarding the Stambak Test for our LF participants is 1.1, which is considered to be large (especially due to the noticeable improvement recorded in R4’s performance). The lower scores of the LF participants in comparison to the HF subjects in ref. [8] in this test are most likely due to the lower cognitive ability of the participants. It should be noted that R3 and R4’s zero scores in the Baseline are because of their lack of cooperation with the human teacher in the Stambak Test. This issue is briefly discussed in Section 5.

In contrast to ref. [8], the reading music phrases test could not be conducted by the human teacher to assess their learnt music knowledge because our LF subjects (unfortunately) could not complete all the scheduled items due to low cognitive skills.

4.1.2 Assessing the social and cognitive skills of the participants

To assess the social and cognitive skills of the subjects, based on the ESCS and common imitation tasks conducted for children with ASD in ABA interventions, 30 tasks were selected/performed three times (i.e. Baseline, Pre-Test, and Post-Test) for the participants by a child psychologist. The conducted items included instruction perception, turn-taking, social reciprocity, gaze-shifting, JA, and gross and fine imitation skills of the participants. Then, after video coding and movie analysis of the developmental assessment sessions, the success rate of each child in all imitation and JA tasks were scored as the children’s performance. Figure 5b–d present the scores of Imitation, RJA, and IJA in the three developmental assessment sessions (i.e. Baseline, Pre-Test, and Post-Test). Figure 5b–d indicate that all participants showed improvement in Imitation, RJA, and IJA.

The imitation scores were positively Pearson’s correlated with the subjects’ overall RJA scores at the average baseline, i.e. the mean of the Baseline and Pre-Test (Pearson correlation coefficient r = 0.98, p = 0.01), and the Post-Test (r = 0.99, p = 0.01) sessions. In addition, the participant’s Imitation performances were also positively associated with the IJA scores at the average of the Baseline and Pre-Test (r = 0.99, p < 0.001), and the Post-Test (r = 0.99, p < 0.001) sessions. Moreover, exploring the relationship between the RJA overall scores and IJA performances indicated strong positive relationships at the average baseline (r = 0.98, p = 0.01) and the Post-Test (r = 0.98, p = 0.02).

4.2 Questionnaires

The participants’ parents filled in four questionnaires four times in the Baseline, Pre-Test, Post-Test, and Follow-up Test during 22 weeks. The results of the questionnaires are presented in Figure 6a–d.

Figure 6

The Questionnaires Scores of the LF subjects in the Baseline (Week #1), Pre-Test (Week #6), Post-Test (Week #14), and Follow-up Test (Week #22) filled in by the parents: (a) overall GARS scores (indicating autism severity), (b) ASSP overall scores, (c) Autism Checklist overall scores, and (d) PSI-SF scores.

4.2.1 GARS

The results of the GARS questionnaire, which indicates the severity of the subject’s autism, are shown in Figure 6a. It should be mentioned that a lower score in this test indicates lower autism severity. This questionnaire covers Stereotyped Behaviors, Communication, Social Interactions, and Developmental Disturbances subscales. As can be seen, slight decreases in autism severity were reported in three out of the four participants from the Pre-Test to Post-Test. However, the amount of change was different for each subject, and only R1 showed noticeable progress.

According to Table 4, the Cohen’s d effect size regarding the overall scores of the GARS for the LF subjects is very small (i.e. 0.19 < 0.2). However, a noticeable point from this questionnaire is that the calculated effect size in the Stereotyped Behaviors subscale is 0.65 (which is close to being large). This result is similar to the findings of ref. [8,16] (e.g. we noticed a large Cohen’s d in this subscale in ref. [8]). In contrast to ref. [8], in which we indicated a large effect size on the Communication subscale for the HF subjects, in this study, the equivalent Cohen’s d was small (Table 4).

It should be noted that the questions regarding the fourth subscale in the GARS (i.e. Developmental Disturbances) are about the first 36 months of the child’s life; therefore, considering the age of the participants in this study, no changes could be seen in this subscale for the subjects.

4.2.2 ASSP

The results of the ASSP questionnaire are presented in Figure 6b. This questionnaire covers three subscales: Social Reciprocity, Social Participation/Avoidance, and Detrimental Social Behaviors. The figure shows that according to the parents’ viewpoint, the social skills of three out of four participants improved from the Pre-Test to Post-Test with the approximate large effect size (i.e. 0.62 > 0.5). For R1, no changes were reported by the parent regarding the ASSP questions. In ref. [8], the calculated Cohen’s d, regarding the overall ASSP scores, was medium (i.e. ∼0.5).

An overview on the ASSP’s subscales indicates that:

1. The Social Reciprocity scores for all of the subjects improved (with a change in mean and standard deviation of 5.0 and 1.14, respectively). The Cohen’s d effect size regarding this subscale is large (>0.8).

2. The Social Participation had a small change (with a change in mean and standard deviation of 1.0 and 0.8, respectively). In addition, a small effect size was calculated for this subscale, which might show that the designed protocol has only a small effect on the Social Participation of children with LF ASD.

3. The Detrimental Social Behaviors was approximately unchanged for the participants. The calculated effect size is near zero for this subscale. Similar to Social Participation, the designed games did not have a measurable change on the detrimental social behaviors of the LF subjects in this study.

Comparing ASSP scores between the Post-Test and Follow-up Test showed us the retention and persistence of the robot-assisted interventions’ effects on the participants.

Imitation performances of the participants were positively associated with the ASSP overall scores at the average baseline, i.e. the mean of the Baseline and Pre-Test scores (Pearson correlation coefficient r = 0.99, p < 0.001) and the mean of the Baseline and the Post-Test scores (r = 0.96, p = 0.04) sessions. The ASSP total scores were also positively Pearson’s correlated with the subjects’ overall RJA scores at the average baseline (r = 0.99, p = 0.01) and marginally correlated with the Post-Test (r = 0.93, p = 0.07) sessions. In addition, exploring the relationship between the ASSP overall scores and IJA performances indicated a strong positive relationship at the average baseline (r = 0.99, p < 0.001) and the Post-Test (r = 0.97, p = 0.02).

4.2.3 Autism checklist

We present the results of the Autism Checklist questionnaire in Figure 6c. This questionnaire includes “Verbal, Language and Communication,” “Socializing,” “Sensory or Cognitive Awareness,” and “Physical and Behavioral Health” subscales. It indicated that the scores of all participants in this questionnaire improved (even if only slightly) in the course of the program. The change in the mean is 5.5 for the subjects of this study. The parents of R1 and R3 reported improvement in their children for more than eight items on this questionnaire. According to Table 4, the Cohen’s d effect size regarding the overall scores of this questionnaire is large (i.e. 0.90 > 0.8). It should be noted that there are only three choices to answer the questions of the Autism Checklist questionnaire, which makes it insensitive to small changes in the symptoms/behaviors of the children.

A close look at the items and subscales of the Autism Checklist questionnaire gives us the following information:

1. In the “Verbal, Language, and Communication” subscale, on average, the LF subjects showed improvement in two items (i.e. mean changes: 1.75). The effect size of this subscale is close to being large (i.e. 0.76 which is ∼0.8).

2. For three out of four participants, the scores of the “Socializing” subscale improved slightly during the intervention sessions (mean changes: ∼1). Regarding the second subscale, the Cohen’s d effect size is small.

3. Based on the parents’ reports, the effect size in the “Sensory or Cognitive Awareness” subscale is also small.

4. The scores of the “Physical and Behavioral Health” subscale had an approximate medium effect size (i.e. ∼0.5).

4.2.4 PSI-SF

In order to investigate the impact of the robot-assisted interventions on the stress level of the participants’ parents, the PSI-SF questionnaire was filled in by the parents four times (i.e. Baseline, Pre-Test, Post-Test, and Follow-up Test) (Figure 6d). It could be seen that in two out of the four participants (i.e. R3 and R4), the parenting stress decreased during the course of the program. According to Table 4, the Cohen’s d effect size regarding this questionnaire is large. It should be noted that in ref. [8], the calculated effect size was medium.

5 Discussion

In this section, the presented results and graphs of the developmental assessments, including the Stambak Rhythm Test, ESCS/imitation, and the questionnaires, are discussed in order to investigate the impact of the proposed robot-assisted interventions on the social and cognitive skills of the participants with LF ASD. The preliminary answers to the research questions of this study are presented. Due to the small number of subjects in this study, the findings of this research should be considered as only an estimation of using this type of technology for the education/rehabilitation of children with LF ASD and cannot necessarily be generalized to all situations for individuals with LF autism.

6 Limitations and future work

We would like to mention that our team tried their best to control the conducted robot-assisted program; however, there is still a possibility that some of the children’s reported improvement items were attributable to other educations/interventions our subjects may have received outside of our program (that were impossible to be fully controlled). Unfortunately, we did not have a control group to determine if the findings of this study were due to our interventions or not. Therefore, as with similar studies [16,26,53], some of the reported children’s improvements could also be due to other educations/treatments received outside our classes (e.g. parents might, even unconsciously, provide further informal opportunities for their children to improve in the specific skills) that were uncontrollable by us. It should be noted that even the children’s maturation during the study might have affected some of the obtained results. Moreover, the small number of participants was the other main limitation of this study. Increasing the number of children as well as the number of conducted sessions could give us more exact information regarding the impact and potential of such robot-assisted sessions in autism treatment. In our study, we would suggest that the reported results should be considered as exploratory findings. Additionally, the quantitative video coding results from the intervention sessions could not be completed during the Covid-19 situation, so only the qualitative descriptions of the children’s performances during the program were presented. In this stage, the effects of the “robot’s attendance” and the “music-based scenario” are not independently detectable in the children’s reported improvements, and the presented results are considered to be based simultaneously on the robot’s attendance and music games. The noticeable amount of unpredictable and maladaptive behaviors of the participants with LF autism sometimes affected the impact of the sessions (which caused some differences between the findings of this study and our previous work [8]). We would like to encourage other scientists to perform such active music-based protocols (even without the robot) to obtain more details about the potential for positive impacts on children with autism. It should be noted that we are still unable to discuss how and if performing the same program without a social robot and only a human mediator would affect (more/less) the conducted sessions for the children with LF autism; therefore, exploring the entire and exact effects of such robot-assisted music-based interventions still needs further complementary studies.

7 Conclusion

In this study, a proposed robot-assisted music-based intervention program was conducted in a rich communicational–educational environment for four children with LF ASD. The protocol included hierarchical tasks with an emphasis on imitation, JA, and turn-taking games. After a 5-month music education program, we found a large effect size in the rhythm reproduction test, which could show that social robots do have the ability to teach music rhythms to children with LF ASD. Unfortunately, due to the deficits in the participants’ cognitive skills alongside the short time interval of the program, no systematic learning of notes and musical phrases occurred by the children in this study. Moreover, we found that the stereotyped behaviors of our LF participants decreased with an approximate large Cohen’s d effect size. In addition, during the developmental assessments, we indicated some improvement in imitation and JA skills of the LF autism participants, which was discussed and compared to the findings of similar papers. The last finding of the article was the decrease in the stress levels of the children’s parents with an approximate large effect size. All in all, we believe that while there is positive potential in using social robots in intervention sessions with children with autism, one should avoid expecting large improvements and consider the reported findings of this study as preliminary exploratory results.